Shell test harness

Dave Fugate dfugate at
Thu Jul 7 11:24:13 PDT 2011

 Glad to hear you're interested in this Paul!  I've heard through the grapevine that Google might have done something in this area (e.g., see, but  I don't know if they're still using it daily or if it works with the latest test262 sources.  Regardless, it hasn't been contributed to test262.

To answer a few of your specific questions:

- the 'negative' property in a globally scoped test being defined implies the test case is expected to throw an exception of some form.  In this case the value, '.', refers to a regular expression matching the exception's message as reported by window.onerror.  Now you're probably asking yourself two questions - why not match the exception type (e.g., SyntaxError) or use a more precise RegExp than '.' (e.g., "parse error")?  The problem here is three-fold.  First, the spec for window.onerror does not require that the exception's type be reported in the message passed to the error handler.  Secondly, ES5 does not require any specific error message for exceptions.  Finally, browser implementers really do need the ability to validate exact error messages even if this shouldn't be done on test262.  As a result, the test262 harness supports validating exact error messages, but the RegExp pattern '.' essentially allows *any* error message

- as David already mentioned, excludelist.xml contains a list of tests that are either disabled due to conflicts with the spec or because there's some issue in test262 infrastructure preventing them from being correctly run.  This file is used only by tools\TestCasePackager\ when generating the website.

- all tests are in test\suite\sputnik_converted and test\suite\ietestcenter.  In the long term, you'll be able to find everything under test\suite\* and there will be no differentiation between the two.  Also, ignore test\suite\sputnik which will be going away soon and has effectively been replaced by external\contributions\Google\sputniktests

- on $ERROR and $FAIL; not sure what the original Sputnik versions of these did, but the test262 versions are identical.  See website\resources\scripts\global\sputnikLib.js.  On a slightly related note, I've been planning on integrating sputnikLib.js into sta.js for some time and make some other small improvements in this area.  A great example of this is decoupling sth.js from jQuery and the HTML entirely, and setting up event handlers for performing actions on passes and fails.  Have a few of these changes in my local enlistment, but nothing of commit quality yet

- last but not least, by all means feel free to create a README:)  Alternatively, I've setup test262 FAQ on the ECMAScript Wiki - see

Thanks for your involvement!


-----Original Message-----
From: test262-discuss-bounces at [mailto:test262-discuss-bounces at] On Behalf Of Paul Biggar
Sent: Wednesday, July 06, 2011 5:27 PM
To: test262-discuss at
Subject: Shell test harness

Hi folks,

Has anyone done any work into making a shell test harness for test262?
I'm interested in adding them to Mozilla's continuous integration, and it would be easier with a shell harness than without.

Assuming there is no harness (I took a brief look around), what do I need to take into account to make one? Some specific questions:

-  I found SputnikGlobalscope.js, which has a format like this:

GlobalScopeTests["S7.2_A5_T1"]={"assertion":"White space cannot be expressed as a Unicode escape sequence consisting of six characters, namely \\u plus four hexadecimal digits","description":"Use TAB (U+0009)","id":"S7.2_A5_T1","path":"07_Lexical_Conventions\\7.2_White_Space\\S7.2_A5_T1.js","negative":"."};

How do I interpret 'negative'? Does '.' mean the test should succeed?

- What does 'config/excludelist.xml' record? Is it the same information in SputnikGlobalscope.js, or is it different?

- Are all the tests in suite/sputnik_converted/, or do I also need to look in ietestcenter/ and sputnik/ ?

- What are the semantics of $ERROR and $FAIL; how do they differ? (I assume $PRINT and $INCLUDE are obvious?)

Thanks in advance. I'm happy to make this into a README to be contributed back if that's useful.


Paul Biggar
Compiler Geek
pbiggar at
test262-discuss mailing list
test262-discuss at

More information about the test262-discuss mailing list