Reorganization of Firefox-UI tests in mozilla-central

Mike de Boer mdeboer at
Mon Sep 5 11:11:49 UTC 2016

>> As I did over IRC, I would like to strongly object to the continued use
>> of per-test-type subfolders in our test directories. You can already use
>> a specific mach command per test type, and the tests are listed in
>> different manifests, *and* there's all the different filename
>> conventions (browser_, test_....html, test_....xul, <whatever>.js) that
>> further point out what type of test you're looking at. The subfolders
>> add nothing useful.
> As someone who has been adding the directory levels to toolkit/components/passwordmgr/test/ recently, I disagree with this since they add a grouping of relevant files making it more obvious which files go with which test suites.
> * Chrome mochitests, plain mochitests and xpcshell share the same prefix (test_) so they are interleaved together in directory listings.
> * Files that accompany tests have no prefix convention.
> * head.js files have no indication of what suite they're for (i.e. no prefix)
> * `mach test` doesn't support specifying a flavor of mochitest.
> * Changing the subdirectory of my `mach mochitest` command is usually faster than adding the flavor argument since the path is usually at the end of the command. Since `mach test` doesn't support the flavor argument I don't have to remember whether to use the argument or change the path as I can always change the path when directories are used.
> * xpcshell and browser-chrome both use "head.js" as the filename for helpers though they run in very different contexts.
> For a new contributor, having each suite in their own directory is much less confusing/overwhelming for the above reasons.
> Password Manager may be special in that it's using four different suites so I'm not suggesting that every component needs to put their tests in a subdirectory named after the test suite but I don't think we should be dumping tests of different suites in one directory unless the distinction between files would be clear to a newcomer.

I think that sub-dividing unit test per-suite/ runner is the wrong way to go, especially for new contributors. It imposes the use and knowledge of jargon that is utterly unnecessary and uninspiring.
Instead, I believe the nicest way to sub-divide tests is by trying (a bit) harder to categorize them better:

NO: /browser/components/sessionstore/test/marionette/
YES: /browser/components/sessionstore/test/startup/

NO: /browser/components/sessionstore/test/xpcshell/
YES: /browser/components/sessionstore/test/cookies/

NO: /any-path/test/general/
YES: /any-path/test/+1-level-up-creative-sub-category/

I suppose that means that I’m strictly opposed to moving the tests, but that’s not the case; having relevant unit tests closer to the metal is a net improvement, whichever way you look at it.
It is, however, I great moment to look up and introspect decisions from the long forgotten past to see if we can do better. (Hint: of course we can! Wouldn’t be much fun otherwise, now would it?)

Let’s file some bugs and get the ball rolling! Also, let’s not invent blockers for Henriks’ work here.

<mini-rant>I don’t quite understand the opposition against unifying the assertion styles and test suite/ runner API surface. Why insist on keeping this separation whereas it’s all written in JS and can be the same with a bit of effort? Same goes for the Python counterpart, of course; let’s get the API surface unified at the very least!</mini-rant>

Hope this helps,


More information about the firefox-dev mailing list