Reorganization of Firefox-UI tests in mozilla-central

Gijs Kruitbosch gijskruitbosch at gmail.com
Fri Sep 2 22:18:26 UTC 2016


None of the test frameworks really cleanly map onto what you understand 
as "unit", "functional", "integration" or "end to end". The frameworks 
are based around capabilities and languages much more than "what kind of 
test would I want to write with this". Those capabilities make them 
maybe more suitable for particular types of tests, but ultimately it's 
the capabilities that determine what kind of test gets written, not 
whether it's a unit or an end to end test.

For instance, this is a marionette test:

http://searchfox.org/mozilla-central/source/browser/components/migration/tests/marionette/test_refresh_firefox.py

This restarts Firefox with some magical commandline flags / env vars and 
checks that the "Firefox refresh" feature behaves more or less sanely.

Likewise, browser mochitests get used for testing units or spec-level 
things depending on the privilege required to easily deduce that the 
spec-level thing has actually happened. For instance, here's a browser 
test that checks which URLs can link to one another based on security 
restrictions:

http://searchfox.org/mozilla-central/source/caps/tests/mochitest/browser_checkloaduri.js

which is pretty much a unit test of scriptSecurityManager, but is a 
mochitest-browser test because creating system- as well as 
content-privileged blob URIs and seeing how they interact with a 
particular other URI isn't easy in any of our other test frameworks.

IOW, I don't think renaming mochitest (and in any case, which one?) to 
"functional" doesn't really help clarify things. We also already kind of 
reached the conclusion that "integration" is a bad name for the fx-ui 
tests, for related reasons.

On 02/09/2016 23:06, Stuart Philp wrote:
> What about renaming mochitest instead, to "functional" or similar? The 
> UI tests that marionette does is in interaction with the UI from a 
> "users" perspective. My understanding of mochitest is it is the 
> interaction with the browser chrome via the api's. Correct?
Not really. mochitest-plain is basically just a web page with some fancy 
semi-privileged APIs. mochitest-chrome is basically just a XUL page with 
some fancy semi-privileged APIs. Both of them use inline or sometimes 
externally-sourced JS to do testing.

mochitest-browser-chrome instead is a JS script that gets run in the 
context of the privileged browser.xul window (ie a normal Firefox 
window) and also uses JS to interact with stuff.

The distinction between marionette or mochitest (of any flavour) calling 
element.click() (or doing so via helpers like synthesizeMouse and 
whatever marionette calls its equivalent of that) seems pretty arbitrary 
to me.***

~ Gijs

*** yes, yes, there are technical differences as to whether your event 
will have the isTrusted flag set to true, but that's mostly not relevant 
for this discussion.



More information about the firefox-dev mailing list