CodeCoverage! Monthly Update

Chris Peterson cpeterson at mozilla.com
Thu Aug 10 20:03:07 UTC 2017


Kyle, do you know if Rust code coverage is blocked on any remaining Rust 
toolchain issues?

chris


On 2017-08-10 11:31 AM, Kyle Lahnakoski wrote:
>
>
> Did you have that sense you were missing something? Well, you were 
> right: You were missing your ...
>
> # *Monthly CodeCoverage! update!  \o/ *
>
> /If you //want to hear more about an//y of the//s//e items, please 
> contact me and I will get you more detailed information/*
> *
>
> *## **Summary of July*
>
>   * *More hard work on the ETL pipeline*: Condensing the data to make
>     it manageable, and writing more efficient code for faster
>     processing. There is still more work to be done, but it's
>     working.  Marco summarized this work with an excellent blog post:
>     [1]
>     https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
>   * *Analyzing coverage variability* - If you recall from previous
>     months, I mentioned that different coverage runs show different
>     coverage numbers: We call this "coverage variability", and it even
>     exists when comparing two runs from the same revision. gmierz has
>     been looking into this variability to better understand its size,
>     and character. [2], [3]
>   * *Finished detailed the plan for the MVP *[4] - This MVP is for
>     visualizing coverage of net-new lines: Relman is interested in the
>     coverage on the net-new lines of every changeset  This is hard
>     given we can only run coverage on every central push. We now have
>     a plan for how to get this done, as best as possible, with the
>     information given.
>
> *## Plans for August*
>
>   * *Build the MVP* - Visualize coverage of net new lines by
>     changeset: Now that we have a plan,  armenzg has already started
>     the frontend UI work [5], [6], and will be working with marco
>     working on the backend [7]
>   * *Continue work on optimizing the ETL pipelines* - necessary work *
>     *
>
> *
> **## Meetings*
>
> We have weekly CodeCoverage meetings, and you are welcome to attend:
>
>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>   * Where: Kyle's video room
>     https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC
>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>
>
> *## Reference*
>
> [1] Blog post on coverage collection and aggregation: 
> https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
>
> [2] Variability analysis code - 
> https://github.com/gmierz/coco-variability-tools
>
> [3] Example variability output - 
> https://gmierz.github.io/variability/variability_index.html
>
> [4] Details for UI - 
> https://public.etherpad-mozilla.org/p/code_coverage_Q3_17
>
> [5] Code for UI - https://github.com/armenzg/code_cov_experiments
>
> [6] Example UI (very rough, but stay tuned!) - 
> https://armenzg.github.io/code_cov_experiments/
>
> [7] Code for backend - 
> https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage
>
>
>
>
> On 2017-07-06 21:37, Kyle Lahnakoski wrote:
>>
>>
>>
>> ## Summary of Past Quarter
>>
>> Coverage is enabled for nearly all tests, and scheduled every push [1]!!
>>
>>   * All c/c++ test suites have coverage enabled
>>   * talos coverage is enabled
>>   * jsvm[7] coverage is enabled, and running
>>   * codecov.io [2] shows the results, broken down by directory
>>
>> ## Plans for Q3
>>
>> The overall plan for Q3 is laid out in the planning document [12].  
>> Put simply, a **coverage differential viewer**, operating at low 
>> resolution (per central push), has enough promise to justify Q3 
>> effort on CodeCoverage.
>>
>> ## The Complications
>>
>>   * Rust code coverage is still delayed [6] - maybe by mid quarter
>>   * The data volume is very large; coveralls.io and codecov.io are
>>     having some difficulty dealing with the volume.
>>   * There is instability in the coverage numbers due to variability
>>     in our test runs; think GC and telemetry logic.  Multiple
>>     coverage runs will be required to get a total coverage number
>>   * Intermittents are impacting coverage reliability - we will
>>     require a coverage health monitor to know if coverage is "complete"
>>
>> ## Summary of this past June
>>
>>   * upgrading tests to use Ubuntu 16.04
>>   * fixing blockers that stood in the way of rust coverage[6]
>>   * enabling coverage to even more suites, like talos [10]
>>   * adding JavaScript to the codecov/coveralls report
>>   * getting a handle on the volume of data code coverage is producing
>>
>> ## Plans for July
>>
>>   * continued work on ETL pipeline
>>   * enable coverage for spidermonkey [11]
>>   * see the first hints of Rust coverage
>>   * build a coverage health monitor to deal with "the complications"
>>     (above)
>>
>> ## Meetings
>>
>> We have weekly CodeCoverage meetings, and you are welcome to attend:
>>
>>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>>   * Where: Kyle's video room
>>     https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC
>>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>>
>>
>> ## Reference
>>
>> [1] See coverage on TH 
>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov
>>
>> [1b] Example on TH: 
>> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=433c379d6e448dc64df25a6b4d8739c99e84d102&filter-searchStr=ccov 
>>
>>
>> [2] codecov.io https://codecov.io/gh/marco-c/gecko-dev
>>
>> [3] Local Coverage 
>> https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build 
>> <https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build>)
>>
>> [4] iOS Coverage 
>> https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master
>>
>> [5] User Cases 
>> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4- 
>>
>>
>> [6] Rust coverage 
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1335518 and 
>> https://github.com/rust-lang/rust/pull/42433
>>
>> [7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174
>>
>> [8] e10s coverage example: 
>> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51
>>
>> [9] Everything: 
>> https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393
>>
>> [10] talos coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1372324
>>
>> [11] spidermonkey coverage: 
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1362013 
>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1372324>
>>
>> [12] Planning document: 
>> https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit?usp=sharing 
>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1372324>
>>
>>
>>
>>
>>
>
>
>
> _______________________________________________
> firefox-dev mailing list
> firefox-dev at mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20170810/7c96086e/attachment-0001.html>


More information about the firefox-dev mailing list