Single test per file or not?
david.bruant at labri.fr
Wed Jul 27 05:02:06 PDT 2011
[+Rick Waldron, because he had a discussion on this topic with Dave
Fugate on Twitter iirc]
Le 27/07/2011 13:28, Geoffrey Sneddon a écrit :
> While the current test262 runner makes the assumption that there is
> only one test per file (see the implementation of
> ES5Harness.registerTest), the WebWorker-based demo MS showed off a
> while back allowed multiple tests per file. Seeming both are, as I
> understand it, by the same group of people, this is an interesting
> Is it intended to allow multiple tests per file, or should there be
> limits to one test per file (and hence only one call to
This is an interesting topic. Granularity.
I have myself been a bit annoyed once or twice by this issue. Typically,
I would run the tests, see one failing and trying to see what was wrong.
I can't remember which, but sometimes, the test was testing several
things at once and by doing so, it was harder to track down where the
non-conformance came from. If i recall, a good share of Sputnik imported
tests tend to do this.
I have not seen a rational or guidelines or rules discussing test
granularity and I think it should be done. I think that for the purpose
of a conformance test suite, the ultimate goal of a test should be to be
able to spot instantaneously where the non-conformance issue comes from.
There are two things that can help out:
1) test description
I have noticed that it wasn't always perfectly accurate. I will report
bugs on that as i find time.
2) test granularity
You may disagree and i'd be happy to have a discussion on how tests
should be designed and maybe providing a set of rules/good
On top of my head, I see one problem which is that some tests have
dependency and rely on other parts of the spec to be conformant. So a
failure in a test can be caused by what the test is testing or one of
its "conformance dependency". I have no idea on how to help out with
this issue, but i wanted to pointed out in case other had ideas.
More information about the test262-discuss