Supporting multiple ECMAScript test suites
ecmascript at norbertlindenberg.com
Tue Aug 21 17:54:58 PDT 2012
On Aug 20, 2012, at 23:27 , Bill Ticehurst wrote:
> Is there an expected status of the specification, quality/coverage level of the test-suite, or level of browser implementation we expect before promoting test suites to the front page of the website?
Testing browsers while they're being updated to meet new specifications is the main purpose of this site, so the level of browser implementations shouldn't play a role here.
I think it would be good to start implementing test cases as soon as portions of the specifications become stable. Over the last six weeks, the discussion of the Internationalization API spec has been largely driven by my work on implementing test cases, and Nebojša's work on bringing the Chrome implementation into conformance, and it has helped uncover a number of issues. In some cases, this results in changes to the spec, and that's a good thing.
This means we do have to clearly document the status of test suites and the underlying specifications, from "partial tests for a specification under development" to "comprehensive test suite of an approved standard". Of course, we know that even the latter will never be complete.
Quality of the test cases is important, so I'm glad you're reviewing my test cases :-)
> The "best practices" suite only has 20 test-cases, with most of these being very narrow in scope (13 of them related to blocks where you can/can't declare a function). Unless this is going to receive some investment, I'm not sure there is value in promoting it to prominence as a separate test suite on the Welcome page.
OK, I agree.
> Likewise for ES6. While we've been iterating on open issues rapidly recently, there have been no submits to the Harmony test-case repository for 9 months, and then only tests for one feature (proxies). Do you see value in promoting this suite to the Welcome page before the spec stabilizes and/or there are more contributions?
We have a Harmony test-case repository? I didn't know! Can you add information on how to access it to the test262 wiki?
I used future tense when talking about the ES6 test suite, but I think serious work on it could start in the coming months, and maybe we'll have something worth listing on the welcome page by the end of the year.
> Obviously Intl402 is a different story, as we are close to final draft and you have a number of tests with broad coverage waiting in the pipeline. Based on your write-up below, I'm not clear how (or if) we would distinguish between test suites for finalized standards (e.g. ES5.1), and standards yet to be finalized (e.g. Intl402). I assume we'd want to make this distinction clear?
> - Bill
> -----Original Message-----
> From: test262-discuss-bounces at mozilla.org [mailto:test262-discuss-bounces at mozilla.org] On Behalf Of Norbert Lindenberg
> Sent: Monday, August 20, 2012 11:58 AM
> To: Rick Waldron; test262-discuss
> Subject: Re: Supporting multiple ECMAScript test suites
> Any other comments on this proposal, or shall we go ahead?
> Rick, currently there's quite a bit of information on contributing on the pages:
> - A section "Who creates and maintains test262?" on the test262 page http://test262.ecmascript.org
> - The "Development" pane of that page
> - Sections "Test Submissions" and "Community Contributions" on the test262 wiki:
> We'd certainly maintain this information in a restructuring of the web site. It probably could be improved, maybe in conjunction with Dave's work on making the wiki more community-friendly. Would you like to try?
> On Aug 13, 2012, at 10:32 , Rick Waldron wrote:
>> On Monday, August 13, 2012 at 1:05 PM, Norbert Lindenberg wrote:
>>> test262 isn't a single test suite anymore: Right now there are three different test suites in the repository - ECMA-262 conformance, ECMA-262 best practices, and ECMA-402 conformance. Soon we may want to fork off an ES6 test suite as well.
>>> What should the web site look like that provides access to three or
>>> more different test suites? Right now, you see
>>> http://test262.ecmascript.org and you don't see
>>> I propose we rename the site to
>>> and then have
>>> as a welcome page with links to:
>>> Later we'll add
>>> which will eventually become the new
>>> What should the corresponding bug database buckets look like? Right
>>> now, there's a
>>> Test262 | Test suite for Ecma-262 implementations product with
>>> components Test Harness | The infrastructure for running test262
>>> Test262 website | The test262.ecmascript.org website Tests | The
>>> individual tests that make up the test262 suite
>>> I propose we change this to
>>> Test Suites | Test suites for ECMAScript implementations Test Harness
>>> | The infrastructure for running tests, including the
>>> tests.ecmascript.org web site
>>> ECMA-262 Tests | The individual tests for ECMA-262 (Language
>>> Specification) conformance
>>> ECMA-402 Tests | The individual tests for ECMA-402
>>> (Internationalization API) conformance
>> My only comment is that of support for this re-org.
>> One thing to keep in mind is making sure complete and up to date instructions for contribution are immediately available on each of the /test-*** pages.
> test262-discuss mailing list
> test262-discuss at mozilla.org
More information about the test262-discuss