Where to put ES6 test cases?
Brian.Terlson at microsoft.com
Wed Apr 17 12:53:39 PDT 2013
Very useful list, thanks for pointing it out!
I agree that using test case markers isn't very ideal. I guess for tests like S188.8.131.52_A1_T01 we would mark it as @negative-es6 (if it's specifically testing that Number.prototype is a number instance) or update the test to be ES6 compliant. You're also right about @positive-es6 (or, alternatively, @negative-es5).
Any thoughts on branching in Mercurial instead? Probably for ES6draft you would only care about master and would see all the tests in the excludelist.xml linked below be removed or updated and wouldn't have to update any custom harnessing...
From: André Bargull [mailto:andre.bargull at udo.edu]
Sent: Wednesday, April 17, 2013 12:38 PM
To: Brian Terlson
Cc: test262-discuss at mozilla.org
Subject: Re: Re: Where to put ES6 test cases?
Some tests only need minor modifications to be ES6 compliant, marking the complete test "@negative-es6" is probably not the right solution.
For example S184.108.40.206_A1_T01 calls right at the beginning `Number.prototype.toString()` and then proceeds with other tests for `Number.prototype.toString`. But the initial call `Number.prototype.toString()` fails since ES6 draft rev.14, because `Number.prototype` is no longer a valid Number instance. That means only that one call to `Number.prototype.toString()` needs to be removed from
S220.127.116.11_A1_T01 to restore ES6 compliance.
And tests like S12.1_A4_T2 actually need a "@positive-es6" marker.
A complete list of test262 test cases which fail in ES6 draft rev.14 can be found here , if you're interested.
> Hi all,
> I would like to get the repository in shape for accepting test cases for new ES6 functionality. There are a couple complexities to deal with. First, it would be good to preserve a clearly identifiable ES5 subset for implementers and so we can continue to fix bugs in ES5 tests while ES6 stabilizes. Second, there is the spec refactoring to consider - most algorithms have been modified and entire sections have been moved. Third, we should come up with a plan that will work for ES7 and beyond as well!
> I brought up this topic a few months back at a TC-39 meeting and there seems to be at least two ways we could go about this: copying existing tests into an ES5 directory and leave them as is while creating an ES6 directory for new test cases, or branching the repository as it is today, call it ES5, and continue working on ES6 branch.
> The first approach is probably easiest, but doesn't have a good way to deal with tests for breaking changes (valid ES5 tests that should fail in ES6 implementations), and doesn't have a good way to organize the ES5 tests into their new ES6 sections. The former limitation could be easily addressed by adding a new attribute to tests that sets expected behavior for ES5 and ES6 (something like @negative-es6).
> The branching approach seems more flexible in that we can make whatever changes we want in the ES6 branch and still have the flexibility of porting ES5 test case fixes back to the ES5 branch (Mercurial seems to be capable of doing merges of files that have moved, but someone please correct me if I'm wrong).
> I'll probably create the ES5 branch later this week unless there is another approach that would be better. Let me know your thoughts...
More information about the test262-discuss