In order to facilitate community work
Mark S. Miller
erights at google.com
Mon Feb 28 08:48:57 PST 2011
On Mon, Feb 28, 2011 at 5:26 AM, David Bruant <bruant at enseirb-matmeca.fr>wrote:
> Here are a couple of suggestions which, in my opinion, would help the
> community work:
> - test262.ecmascript.org
> -- In the Results page, it would be useful to have a link to the test file
> in the http://hg.ecmascript.org/tests/test262/ website (may help to spot
> differences between the run test suite and the latest )
> -- A "report a bug" link next to each bug result (or in the "source"
> popup?) would be useful too. If there is such a feature in Bugzilla, the
> link could carry a preformatted bug title and pre-fill the bug report to
> point to the "Tests" component. Uniform bug titles make text-search easier.
> - Test searching
> There are now about 10000 tests. By default, they are sorted in the repo by
> source(Sputnik | ietestcenter)/chapter/section/subsection/...
> Even if it certainly makes sense from a test suite producer point of view
> (and directory choices has to be made anyway), it doesn't cover all search
> use cases. In order to "identify test holes" more efficiently as suggested
> under "Community Contributions" it would be helpful to have some tool to
> search for tests. For instance, it's currently hard to find if a test has
> been forgotten regarding:
> * native prototype objects
> For instance, in 126.96.36.199 is written "The [[Prototype]] internal property
> of the newly constructed object is set to the original Array prototype
> object, the one that is the initial value of Array.prototype". There is
> clearly a test to write based on that to make sure that Monkey-patched
> Array.prototype aren't used. Such a pattern can be found for functions
> (15.3.3, 15.3.4), objects (188.8.131.52 step 4) and certainly in other places.
> * length properties
At <http://codereview.appspot.com/4182070/> you'll see a draft data-driven
test generation framework I'm working on, where most of the tests will be
generated from a JSON description of what should be the initial state of a
conforming ES5 heap (at stdheap.json). Currently the test driver at <
http://www.erights.org/tests/testjs/> just does the described tests directly
(try it). As Christian suggests in his comments, I'm going to rework this to
generate the individual tests as separate files, so that it will integrate
well into existing test frameworks.
My three biggest requests for test262:
* Be able to run offline, as sputnik and es5conform can, so testing can be
used as part of a normal development cycle.
* Be able to build test262 from test262 sources in a portable manner, with
depending on MS-specific C# and powershell code.
* Rework the test harness so that sputnik tests can convert into something
similarly terse and maintainable. Right now, the sputnik->test262 conversion
does not produce tests maintainable as sources.
Until these are addressed, I'll be adding further tests to sputnik and
re-auto-converting to test262 periodically. Once these are addressed, we'd
like to do a last conversion and then maintain the test262 sources as
> * Errors (make sure that code that should throw errors have a test for
> * strict mode (I assume it's not going to be covered in a chapter)
> * Any other cross-chapter topic you could think of.
> Several different approaches could be taken to tackle that issue:
> -- text-search
> -- tags search
> - Run partial test suite
> Running the test suite is currently long. I'm currently reviewing FF4
> fails. When I close my web browser, I have to re-run all tests. However,
> most of the time, I don't care about most of them. I just want all those
> that I haven't reviewed yet. I wish the website could offer an easy way to
> run for instance:
> - all chapter 15.2 tests (or any chapter/section/subsection obviously)
> - all tests found through a search (if/when some search feature will be
> - all tests that has been added or changed between two test suite versions.
> If I have reviewed all tests of version x, I may not care about them when
> the x+1 version comes out.
> - Accept user contributed tests
> I have read that "Ecma’s intellectual property policies, permit only Ecma
> members to directly contribute code to the project.". I however think that
> this test suite would benefit from accepting contributions from the
> programming language. I think there is a need to open up the way tests can
> be contributed.
> If the Ecma’s intellectual property policies cannot be changed/adapted to
> make that happen, could ECMA TC-39 members find a way to accept user
> contributions, review them and then submit them to Ecma? For instance, by
> creating a TC-39 github account and a test repo, people create tests, submit
> them to this repo and when there are enough of them or after a certain
> amount of time, TC-39 members officially submit the tests to ECMA. There
> are certainly intellectual property/copyright issues that has to be
> considered, by I'm drawing some sort of main idea.
> I'd be happy to help out if there is anything I can do to help with
> test262-discuss mailing list
> test262-discuss at mozilla.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the test262-discuss