Where to put ES6 test cases?

Norbert Lindenberg ecmascript at lindenbergsoftware.com
Wed Apr 24 14:11:19 PDT 2013


On Apr 24, 2013, at 12:56 , Brian Terlson wrote:

>> -----Original Message-----
>> From: Norbert Lindenberg [mailto:ecmascript at lindenbergsoftware.com]
>> 
>> On Apr 18, 2013, at 11:34 , Brian Terlson wrote:
>> 
>>>> -----Original Message-----
>>>> From: Allen Wirfs-Brock [mailto:allen at wirfs-brock.com]
>> 
>>>> I agree that identifying tests by section/algorithm/step number is proving
>> to
>>>> be problematic.  The basic thing we were trying to accomplish was to have
>> a
>>>> way to track coverage of the specification's many requirements.  I still
>> think
>>>> some sort of coverage tracking at this level is important.  Rather than
>> baking
>>>> it into the test names perhaps we could use some other sort of scheme
>> such
>>>> as a coverage data base where we associate tests with requirements that
>>>> they validate.
>>> 
>>> [Brian Terlson] The biggest pain IMO was simply having the ID baked into
>> the file name, which also doesn't seem to be desirable in light of the spec
>> refactor (you could end up with conflicts between ES5-named tests and ES6-
>> named tests). I agree we should continue being as specific as practical when
>> developing and contributing tests. I was thinking of persisting what's
>> presently in the filename as an attribute in the test file, such as @es5id, and
>> add a new @es6id attribute as well. The ID MAY be present, and if present
>> MUST specify a spec section and MAY specify a specific algorithm step.
>> Depending on what you had in mind for the coverage database this might be
>> functionally identical to what you were proposing... thoughts?
>> 
>> I would actually try to remove spec sections from directory names, file
>> names, and attributes as much as possible. At least for the current clause 15,
>> I would find
>>   Object/func
>>   Object/cons
>>   Object/consprops/prototype
>>   Object/consprops/getPrototypeOf
>>   Object/protprops/constructor
>>   Object/protprops/toString
>>   String/instprops/length
>> far more useful than any section numbers - and the names are guaranteed to
>> be stable as well.
>> 
>> The individual tests then only need names identifying them within their
>> respective algorithms - that could be a descriptive word, just a number, or
>> step numbers if it's clear which spec version they come from.
>> 
>> Abstract operations don't have that stability guarantee, but even for them
>> names are more meaningful than section numbers.
> 
> I think a better named directory structure is a great idea! I really like your basic proposal (at least for chapter 15 it makes a ton of sense, but might be harder for the others). I'm also a fan of just naming the test case files some serial number so as to not waste time coming up with names for 100 test cases that are testing roughly the same thing, but I suppose people can be free to name their test case files whatever they want. Unless anyone objects I think we should do this reorganization.
> 
> However, I am not sure that removing all links to spec sections is a good idea. I think it should be optional (I can see sometimes we may want to "dump" a bunch of test cases that are useful without having to go through and tag each one), but there should be a way we can get a rough idea of the coverage for any given spec section or algorithm. I'm not sure how well this would work if we didn't do the extra effort of tying test cases to spec sections. Thoughts?

There could be a file somewhere mapping directories to sections:
  Object/func: 15.2.1
  Object/cons: 15.2.2
  Object/consprops/prototype: 15.2.3.1
  Object/consprops/getPrototypeOf: 15.2.3.2
  Object/protprops/constructor: 15.2.4.1
  Object/protprops/toString: 15.2.4.2
  String/instprops/length: 15.5.5.1
Whatever tool we use to check coverage would use this file - no need to bulk up all the tests with this information. Only information that's specific to a test (such as the steps being tested) should be inside that test. We may be able to extract the mapping automatically from the table of contents of the spec.

BTW, before we reorganize anything, we have to get rid of the path attributes in 11590 files.

Norbert


More information about the test262-discuss mailing list