Reorganization of Firefox-UI tests in mozilla-central

Gregory Szorc gps at mozilla.com
Mon Sep 26 17:50:41 UTC 2016


On Mon, Sep 5, 2016 at 4:11 AM, Mike de Boer <mdeboer at mozilla.com> wrote:

>
> >>
> >> As I did over IRC, I would like to strongly object to the continued use
> >> of per-test-type subfolders in our test directories. You can already use
> >> a specific mach command per test type, and the tests are listed in
> >> different manifests, *and* there's all the different filename
> >> conventions (browser_, test_....html, test_....xul, <whatever>.js) that
> >> further point out what type of test you're looking at. The subfolders
> >> add nothing useful.
> >
> > As someone who has been adding the directory levels to
> toolkit/components/passwordmgr/test/ recently, I disagree with this since
> they add a grouping of relevant files making it more obvious which files go
> with which test suites.
> >
> > * Chrome mochitests, plain mochitests and xpcshell share the same prefix
> (test_) so they are interleaved together in directory listings.
> > * Files that accompany tests have no prefix convention.
> > * head.js files have no indication of what suite they're for (i.e. no
> prefix)
> > * `mach test` doesn't support specifying a flavor of mochitest.
> > * Changing the subdirectory of my `mach mochitest` command is usually
> faster than adding the flavor argument since the path is usually at the end
> of the command. Since `mach test` doesn't support the flavor argument I
> don't have to remember whether to use the argument or change the path as I
> can always change the path when directories are used.
> > * xpcshell and browser-chrome both use "head.js" as the filename for
> helpers though they run in very different contexts.
> >
> > For a new contributor, having each suite in their own directory is much
> less confusing/overwhelming for the above reasons.
> >
> > Password Manager may be special in that it's using four different suites
> so I'm not suggesting that every component needs to put their tests in a
> subdirectory named after the test suite but I don't think we should be
> dumping tests of different suites in one directory unless the distinction
> between files would be clear to a newcomer.
>
> I think that sub-dividing unit test per-suite/ runner is the wrong way to
> go, especially for new contributors. It imposes the use and knowledge of
> jargon that is utterly unnecessary and uninspiring.
> Instead, I believe the nicest way to sub-divide tests is by trying (a bit)
> harder to categorize them better:
>
> NO: /browser/components/sessionstore/test/marionette/
> YES: /browser/components/sessionstore/test/startup/
>
> NO: /browser/components/sessionstore/test/xpcshell/
> YES: /browser/components/sessionstore/test/cookies/
>
> NO: /any-path/test/general/
> YES: /any-path/test/+1-level-up-creative-sub-category/
>
>
> I suppose that means that I’m strictly opposed to moving the tests, but
> that’s not the case; having relevant unit tests closer to the metal is a
> net improvement, whichever way you look at it.
> It is, however, I great moment to look up and introspect decisions from
> the long forgotten past to see if we can do better. (Hint: of course we
> can! Wouldn’t be much fun otherwise, now would it?)
>

Naming things is hard and one of the more difficult problems in computer
science.

You raise valid points that our testing is overly complicated, with N
variations of tests and corresponding ways to name them.

>From a UX perspective, you have to pick the audience you are optimizing
for. If you optimize for running tests, you probably put everything in one
giant bucket/directory and/or lean on the tagging system in test manifests
to logically group related tests to make them easier to run. Then you just
tell people "run `mach test my-feature` or `mach test
path/to/my/component`" and magic ensues. If you optimize for writing and
maintaining tests, then you start wanting a naming convention.


>
> Let’s file some bugs and get the ball rolling! Also, let’s not invent
> blockers for Henriks’ work here.
>
> <mini-rant>I don’t quite understand the opposition against unifying the
> assertion styles and test suite/ runner API surface. Why insist on keeping
> this separation whereas it’s all written in JS and can be the same with a
> bit of effort? Same goes for the Python counterpart, of course; let’s get
> the API surface unified at the very least!</mini-rant>
>

You are preaching to the choir. When I was on the Firefox team, I found the
N variations for writing tests to be maddening. The variation adds
significant cognitive overhead. My personal experience is that unless the
underlying code being tested is in a more verbose language (C, C++, Java)
or has significant complexity, it takes longer to write tests than the code
itself (often 2-3x more time). But with Firefox features, I found that test
writing consumed a disproportional amount of time when compared to my
experience writing similar code that wasn't for Firefox. And I chalk a lot
of this up to the complexity of writing tests for Firefox and the lack of a
unified, well-polished set of tools and standards. I view the inconsistent
testing "APIs" used by Firefox a massive source of technical debt and an
ongoing burden on developer productivity. How or who fixes that, I'm not
sure. Furthermore, the last developer productivity survey showed that
people think debugging [intermittent] test failures and things like Try
times to be the most important item w.r.t. test running. So that's where
the focus of the productivity team currently is. I'd love for someone to
take ownership of unifying the test "APIs."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20160926/7999f7ad/attachment.html>


More information about the firefox-dev mailing list