Data platform update

Georg Fritzsche gfritzsche at mozilla.com
Fri Dec 23 15:22:35 UTC 2016


Data platform update

The data platform team is working on our core Telemetry system, the data
pipeline and providing core datasets, with support from the Firefox data
engineering and the Data tools team.

To make new features more visible, we intend to provide quarterly updates,
starting with this one.

What's new in the last few months?

On the data collection side, we added scalars
<https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/collection/scalars.html>
and added engagement measures
<https://www.a2p.it/wordpress/tech-stuff/mozilla/measuring-tab-and-window-usage-in-firefox/>
on top. We now support recording histograms in child processes and
added categorical
histograms
<https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type>
.

We improved documentation, starting from the Telemetry wiki page
<https://wiki.mozilla.org/Telemetry> and updated onboarding
<https://air.mozilla.org/roberto-vitillo-telemetry-and-other-data-sources/>.

Further work happened on:

   -

   improved client-side documentation
   <https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/index.html>


   -

   an introduction on analyzing telemetry data
   <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/analyzing_telemetry_data.md>
   -

   a guide on choosing a dataset
   <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/choosing_a_dataset.md>
   for analysis
   -

   updates to examples for the longitudinal dataset
   <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/longitudinal_examples.md>
   -

   exploring how to make Firefox data more discoverable
   <http://georgf.github.io/fx-data-explorer/>


The data pipeline work powers results for re:dash
<https://sql.telemetry.mozilla.org/> and custom analysis
<https://analysis.telemetry.mozilla.org/> among other things.

Notable recent work here includes:

   -

   the cross-sectional
   <https://github.com/mozilla/telemetry-batch-view/blob/master/docs/choosing_a_dataset.md#cross-sectional>
   dataset is now operational in re:dash
   <https://sql.telemetry.mozilla.org/>
   -

   socorro crash data
   <https://sql.telemetry.mozilla.org/queries/1170/source#table> is now
   available in re:dash
   -

   the new dataset API
   <http://python-moztelemetry.readthedocs.io/en/latest/userguide.html#module-moztelemetry.dataset>
   improves querying raw ping data
   -

   the self-serve analysis portal <https://analysis.telemetry.mozilla.org/>
   was relaunched with improved UX
   -

   provided easy access to self-serve real-time analysis
   <https://gist.github.com/trink/bf7d6e504b3fd937b73095412612ae5a>
   -

   launched a knowledge repository <https://reports.telemetry.mozilla.org>
   to make reports discoverable in one place and easier to review


Coming soon

For the next few months, interesting projects in the pipeline include:

   -

   event telemetry <https://wiki.mozilla.org/Event_Telemetry>, which
   enables recording event data into Telemetry in a common format
   <https://docs.google.com/document/d/1hNuS9lUJMvMqgntZXbFA6xZBU9zBpQgo7x73-sXKRpI/edit#heading=h.62xjq5nppfns>
   -

   work to decrease data latency, allowing us to make decisions faster
   -

   adding Telemetry support for Add-ons
   -

   making it easy for new pings to show up in re:dash thanks to
   direct-to-parquet
   -

   enabling efficient lookup of client histories using Hbase
   -

   improved alerting
   <https://robertovitillo.com/2016/12/15/a-martingale-approach-to-detect-changes-in-histograms/>
   for Telemetry probes
   -

   creating a new crash summary dataset, to make it easier to analyze crash
   data


Contact us


Please reach out to us with any questions or concerns.


You can find us on IRC in #telemetry and #datapipeline.

The main mailing list for data topics is fhr-dev
<https://mail.mozilla.org/listinfo/fhr-dev>.

Bugs can be filed in one of these components
<https://wiki.mozilla.org/Telemetry#Filing_Bugs>.

You can also find us on Twitter as @MozTelemetry
<https://twitter.com/moztelemetry>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20161223/fbc1ee19/attachment.html>


More information about the firefox-dev mailing list