Reorganization of Firefox-UI tests in mozilla-central

Stuart Philp sphilp at mozilla.com
Fri Sep 2 22:06:17 UTC 2016


I'm a bit ignorant here, not overly familiar with how things are done in
m-c, please correct me if I'm off on something.

What about renaming mochitest instead, to "functional" or similar? The UI
tests that marionette does is in interaction with the UI from a "users"
perspective. My understanding of mochitest is it is the interaction with
the browser chrome via the api's. Correct?

I'd also add integration is a bit of a loaded term as although it implies
that you are testing the combination of components, it is more so at a
functional level, which only kind of inherently happens with marionette
tests, it isn't really their primary purpose. They are kind of one level
abstracted from that. Meaning: the unit, code block level -> the
functional, spec level -> the ui, end-user level.

Alternatively, end-to-end (e2e) is a term my team has used to show a test
is approaching things from the "end-users" perspective.

I would also generally say putting tests in sub-directories organized by
type/purpose is nice as the different test frameworks are a bit of a
context switch (possibly in language, api's, methodologies). So for a given
feature you might have:

my-feature/tests/unit
my-feature/tests/functional
my-feature/tests/integration
my-feature/tests/e2e

I understand that is a bit of folder explosion, but it immediately
illustrates the purpose of the files within, as opposed to a type extension
like "test_*_type" which isn't always apparent, or worse a generic "test_*"
prefix which explains nothing until you open the file. The gains around
clarity seem to outweigh the drawbacks?


On Fri, Sep 2, 2016 at 3:08 AM, Henrik Skupin <mail at hskupin.info> wrote:

> Gijs Kruitbosch wrote on 09/01/2016 06:24 PM:
>
> > As I did over IRC, I would like to strongly object to the continued use
> > of per-test-type subfolders in our test directories. You can already use
> > a specific mach command per test type, and the tests are listed in
> > different manifests, *and* there's all the different filename
> > conventions (browser_, test_....html, test_....xul, <whatever>.js) that
> > further point out what type of test you're looking at. The subfolders
> > add nothing useful.
>
> The problematic piece here will be the package-tests step which
> currently picks complete subfolders. It would mean if we mix-up tests
> for firefox-ui-tests and eg. mochitests all would end-up twice in the
> common.tests.zip archive. If we want to get away from using subfolders
> we would have to improve the test archiver
> (https://dxr.mozilla.org/mozilla-central/source/python/
> mozbuild/mozbuild/action/test_archive.py)
> first to not only collect the directories of referenced manifests, but
> also only pick those tests which are referenced and leave all others
> behind. This would apply to all test suites currently covered by this
> mozbuild action.
>
> > Finally, "firefox_ui" (as well as "ui") as a name for a directory is
> > going to cause all kinds of confusion for people exploring the repo
> > without detailed knowledge of what's going on. Additionally, it's not
> > like most of the mochitest-browser tests aren't tests of the Firefox
> > UI... If we absolutely must have some kind of subdirectory because of
> > reasons I have yet to hear, I think "integration" would be a better
> > choice of name as far as subdirs of "test" go (as juxtaposed to "unit"
> > for our xpcshell tests).
>
> I would totally be fine with integration as name, unless we don't hit
> the above mentioned problem that tests for other harnesses end-up in the
> same folder.
>
> --
> Henrik
> _______________________________________________
> firefox-dev mailing list
> firefox-dev at mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20160902/ebc52af3/attachment.html>


More information about the firefox-dev mailing list