Single test per file or not?

Dave Fugate dfugate at microsoft.com
Wed Jul 27 08:28:02 PDT 2011


Hi Geoffrey, thanks for your suggestion, and I'll try to shed a bit of light on why test262 does not follow the "single *.js" test case pattern the WebWorker demo was.  Some fraction of the tests on test262 are destructive in nature.  For example, I recently came across a test that was doing something like calling Object.freeze on the global object.  If we tried bundling test262 files together, this type of test would have to be executed last, there couldn't be similar destructive tests in the same file, etc.  We're not limited to destructive failures in nature either, and I've seen the following class of issues occur:
	//Test A
	function testcase() {
		//adds a "abc" property to the global object; does some verification on it; does not remove "abc" from the global object
	}
	//Test B
	function testcase() {
		//does something that should fail to add "abc" property to the global object; verifies that "abc" isn't there
	}

Running these two tests in the same file fails if:
1. Test B is run first and fails (i.e., "abc" is added to the global object => it doesn't matter if Test A succeeds in doing this or not, the test will pass regardless)
2. Test A runs first and forgets to delete "abc" => Test B cannot fail
3. Test A runs first and "abc" is a non-config/non-writable property => there's no way "abc" can be deleted by Test A to clean-up the execution environment for Test B

The two reasons I gave above are also why only a smallish subset of chapter 15.2 tests were used in the WebWorker demo instead of the entire test262 suite. 

Again, good questions and I'll add these to our new FAQ at http://wiki.ecmascript.org/doku.php?id=test262:faq.

My best,

Dave


-----Original Message-----
From: test262-discuss-bounces at mozilla.org [mailto:test262-discuss-bounces at mozilla.org] On Behalf Of Geoffrey Sneddon
Sent: Wednesday, July 27, 2011 4:29 AM
To: test262-discuss at mozilla.org
Subject: Single test per file or not?

While the current test262 runner makes the assumption that there is only one test per file (see the implementation of ES5Harness.registerTest), the WebWorker-based demo MS showed off a while back allowed multiple tests per file. Seeming both are, as I understand it, by the same group of people, this is an interesting change.

Is it intended to allow multiple tests per file, or should there be limits to one test per file (and hence only one call to ES5Harness.registerTest)?

--
Geoffrey Sneddon - Opera Software
<http://gsnedders.com>
<http://opera.com>
_______________________________________________
test262-discuss mailing list
test262-discuss at mozilla.org
https://mail.mozilla.org/listinfo/test262-discuss



More information about the test262-discuss mailing list