Single test per file or not?

Geoffrey Sneddon gsneddon at opera.com
Sat Jul 30 11:43:05 PDT 2011


On 27/07/11 16:38, Dave Fugate wrote:
> One other thing worth mentioning I nearly forgot - suppose ES6 adds a new 'z' modifier for RegExp objects and accordingly we add some new test case like /a/z.test("a").  If this test case is added to a file containing other test cases *and* browser A does not yet support the 'z' modifier, browser A will end up failing *all* test cases as the entire file is unparsable.  I think with all the syntax changes in store for ES6, this is a very likely scenario.  Also, I saw a variant of the 'z' example above occur during IE9 product development.
>
> My best,
>
> Dave
>
> -----Original Message-----
> From: test262-discuss-bounces at mozilla.org [mailto:test262-discuss-bounces at mozilla.org] On Behalf Of Dave Fugate
> Sent: Wednesday, July 27, 2011 8:28 AM
> To: Geoffrey Sneddon; test262-discuss at mozilla.org
> Subject: RE: Single test per file or not?
>
> Hi Geoffrey, thanks for your suggestion, and I'll try to shed a bit of light on why test262 does not follow the "single *.js" test case pattern the WebWorker demo was.  Some fraction of the tests on test262 are destructive in nature.  For example, I recently came across a test that was doing something like calling Object.freeze on the global object.  If we tried bundling test262 files together, this type of test would have to be executed last, there couldn't be similar destructive tests in the same file, etc.  We're not limited to destructive failures in nature either, and I've seen the following class of issues occur:
> 	//Test A
> 	function testcase() {
> 		//adds a "abc" property to the global object; does some verification on it; does not remove "abc" from the global object
> 	}
> 	//Test B
> 	function testcase() {
> 		//does something that should fail to add "abc" property to the global object; verifies that "abc" isn't there
> 	}
>
> Running these two tests in the same file fails if:
> 1. Test B is run first and fails (i.e., "abc" is added to the global object =>  it doesn't matter if Test A succeeds in doing this or not, the test will pass regardless) 2. Test A runs first and forgets to delete "abc" =>  Test B cannot fail 3. Test A runs first and "abc" is a non-config/non-writable property =>  there's no way "abc" can be deleted by Test A to clean-up the execution environment for Test B
>
> The two reasons I gave above are also why only a smallish subset of chapter 15.2 tests were used in the WebWorker demo instead of the entire test262 suite.
>
> Again, good questions and I'll add these to our new FAQ at http://wiki.ecmascript.org/doku.php?id=test262:faq.

Alright, I was mainly asking while writing wrapper to get tests running 
in our regression tracking system (and what assumptions I could make).

For what it's worth, for the testsuite written for Carakan, tests are 
grouped in files, where each testcase is a function in the global scope 
whose name begins with "test". The global property problem was dealt 
with mostly by virtue of each test being run in a different runtime (it 
was destroyed/recreated for each test being run). Might get around to 
releasing at least some of them at some point (some of them test very 
engine-specific optimizations in ways that give away things we probably 
don't want to have public).

The main reason for having multiple tests per file was to reduce the 
overhead of writing new tests, as writing new test for some feature we 
already have tests for is just a matter of writing a new function within 
that file.

We did have fun with some of the tests written for Futhark, as a number 
of them tested some of the parts of ES4 Futhark supported, which Carakan 
does not, though the structure of those tests was… flawed, in many ways. :)

-- 
Geoffrey Sneddon — Opera Software
<http://gsnedders.com>
<http://opera.com>


More information about the test262-discuss mailing list