CodeCoverage! Monthly Update

Sylvestre Ledru sledru at mozilla.com
Thu Aug 10 20:10:53 UTC 2017


The patch providing coverage in rust landed here:
https://github.com/rust-lang/rust/pull/42433

If I understood correctly, we are waiting for rust 1.20 to be able to
use it.

Sylvestre




Le 10/08/2017 à 22:03, Chris Peterson a écrit :
> Kyle, do you know if Rust code coverage is blocked on any remaining
> Rust toolchain issues?
>
> chris
>
>
> On 2017-08-10 11:31 AM, Kyle Lahnakoski wrote:
>>
>>
>> Did you have that sense you were missing something? Well, you were
>> right: You were missing your ...
>>
>> # *Monthly CodeCoverage! update!  \o/ *
>>
>> /If you //want to hear more about an//y of the//s//e items, please
>> contact me and I will get you more detailed information/*
>> *
>>
>> *## **Summary of July*
>>
>>   * *More hard work on the ETL pipeline*: Condensing the data to make
>>     it manageable, and writing more efficient code for faster
>>     processing. There is still more work to be done, but it's
>>     working.  Marco summarized this work with an excellent blog post:
>>     [1]
>>     https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
>>   * *Analyzing coverage variability* - If you recall from previous
>>     months, I mentioned that different coverage runs show different
>>     coverage numbers: We call this "coverage variability", and it
>>     even exists when comparing two runs from the same revision.
>>     gmierz has been looking into this variability to better
>>     understand its size, and character. [2], [3]
>>   * *Finished detailed the plan for the MVP *[4] - This MVP is for
>>     visualizing coverage of net-new lines: Relman is interested in
>>     the coverage on the net-new lines of every changeset  This is
>>     hard given we can only run coverage on every central push. We now
>>     have a plan for how to get this done, as best as possible, with
>>     the information given.
>>
>> *## Plans for August*
>>
>>   * *Build the MVP* - Visualize coverage of net new lines by
>>     changeset: Now that we have a plan,  armenzg has already started
>>     the frontend UI work [5], [6], and will be working with marco
>>     working on the backend [7]
>>   * *Continue work on optimizing the ETL pipelines* - necessary work *
>>     *
>>
>> *
>> **## Meetings*
>>
>> We have weekly CodeCoverage meetings, and you are welcome to attend:
>>
>>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>>   * Where: Kyle's video room
>>     https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC
>>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>>
>>
>> *## Reference*
>>
>> [1] Blog post on coverage collection and aggregation:
>> https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
>>
>> [2] Variability analysis code - 
>> https://github.com/gmierz/coco-variability-tools
>>
>> [3] Example variability output -
>> https://gmierz.github.io/variability/variability_index.html
>>
>> [4] Details for UI -
>> https://public.etherpad-mozilla.org/p/code_coverage_Q3_17
>>
>> [5] Code for UI - https://github.com/armenzg/code_cov_experiments
>>
>> [6] Example UI (very rough, but stay tuned!) -
>> https://armenzg.github.io/code_cov_experiments/
>>
>> [7] Code for backend -
>> https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage
>>
>>
>>
>>
>> On 2017-07-06 21:37, Kyle Lahnakoski wrote:
>>>
>>>
>>>
>>> ## Summary of Past Quarter
>>>
>>> Coverage is enabled for nearly all tests, and scheduled every push
>>> [1]!!
>>>
>>>   * All c/c++ test suites have coverage enabled
>>>   * talos coverage is enabled
>>>   * jsvm[7] coverage is enabled, and running
>>>   * codecov.io [2] shows the results, broken down by directory
>>>
>>> ## Plans for Q3
>>>
>>> The overall plan for Q3 is laid out in the planning document [12]. 
>>> Put simply, a **coverage differential viewer**, operating at low
>>> resolution (per central push), has enough promise to justify Q3
>>> effort on CodeCoverage.
>>>
>>> ## The Complications
>>>
>>>   * Rust code coverage is still delayed [6] - maybe by mid quarter
>>>   * The data volume is very large; coveralls.io and codecov.io are
>>>     having some difficulty dealing with the volume.
>>>   * There is instability in the coverage numbers due to variability
>>>     in our test runs; think GC and telemetry logic.  Multiple
>>>     coverage runs will be required to get a total coverage number
>>>   * Intermittents are impacting coverage reliability - we will
>>>     require a coverage health monitor to know if coverage is "complete"
>>>
>>> ## Summary of this past June
>>>
>>>   * upgrading tests to use Ubuntu 16.04
>>>   * fixing blockers that stood in the way of rust coverage[6]
>>>   * enabling coverage to even more suites, like talos [10]
>>>   * adding JavaScript to the codecov/coveralls report
>>>   * getting a handle on the volume of data code coverage is producing
>>>
>>> ## Plans for July
>>>
>>>   * continued work on ETL pipeline
>>>   * enable coverage for spidermonkey [11]
>>>   * see the first hints of Rust coverage
>>>   * build a coverage health monitor to deal with "the complications"
>>>     (above)
>>>
>>> ## Meetings
>>>
>>> We have weekly CodeCoverage meetings, and you are welcome to attend:
>>>
>>>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>>>   * Where: Kyle's video room
>>>     https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC
>>>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>>>
>>>
>>> ## Reference
>>>
>>> [1] See coverage on TH
>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov
>>>
>>> [1b] Example on TH:
>>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=433c379d6e448dc64df25a6b4d8739c99e84d102&filter-searchStr=ccov
>>>
>>>
>>> [2] codecov.io https://codecov.io/gh/marco-c/gecko-dev
>>>
>>> [3] Local Coverage
>>> https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
>>> <https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build>)
>>>
>>> [4] iOS Coverage
>>> https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master
>>>
>>> [5] User Cases
>>> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-
>>>
>>>
>>> [6] Rust coverage
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1335518 and
>>> https://github.com/rust-lang/rust/pull/42433
>>>
>>> [7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174
>>>
>>> [8] e10s coverage example:
>>> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51
>>>
>>> [9] Everything:
>>> https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393
>>>
>>> [10] talos coverage:
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1372324
>>>
>>> [11] spidermonkey coverage:
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1362013
>>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1372324>
>>>
>>> [12] Planning document:
>>> https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit?usp=sharing
>>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1372324>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> _______________________________________________
>> firefox-dev mailing list
>> firefox-dev at mozilla.org
>> https://mail.mozilla.org/listinfo/firefox-dev
>
>
>
> _______________________________________________
> firefox-dev mailing list
> firefox-dev at mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20170810/7695c595/attachment-0001.html>


More information about the firefox-dev mailing list