Activity Stream - Post Launch Engineering Update

Tim Spurway tspurway at mozilla.com
Tue Dec 5 21:40:09 UTC 2017


good question, jared.

The bars are for a weekly resolution and they don’t change much from week to week.  I was comparing the data from the most recent week.  

These two graphs compare AS on about:home <about:home> vs. AS on about:newtab <about:newtab> in 57.

We were a bit surprised that we see such a difference between these surfaces.  Maybe the differences are enough to actually differentiate the UX on these two different Firefox pages.  It’s definitely worth running more experiments.


> On Dec 5, 2017, at 4:18 PM, Jared Wein <jaws at mozilla.com> wrote:
> 
> Wow, that video shows great improvement! Nice job!
> 
> As to the engagement gains, what numbers are you comparing? For example, "Highlights (from 1.6% to 2.6%, a +62% gain)". The charts start on November 12 and Quantum was released on November 14th. Are these engagement gains seen between v57 and v58 users?
> 
> Thanks,
> Jared
> 
> On Tue, Dec 5, 2017 at 3:52 PM, Tim Spurway <tspurway at mozilla.com <mailto:tspurway at mozilla.com>> wrote:
> Hey Folks,
> 
> Activity Stream was released as the default about:newtab <> and about:home <> experience for Firefox Quantum just about three weeks ago.  The team is proud to report that Activity Stream is performing well in release!
> 
> Here’s a cool graph from the Activity Stream main dashboard <https://sql.telemetry.mozilla.org/dashboard/activity-stream-system-addon-metrics-summary_1?p_channel=release&p_granularity=week>:
> 
> <image.png>
> 
> This is the engagement rates of the various Activity Stream sections/components for both about:home <> and about:newtab <>.  Note that Activity Stream is an almost identical experience on both pages.  The first thing you notice about this data is that people are using the Home and Newtab pages very differently  -  more people interact with Activity Stream on about:home <> (49%) than on about:newtab <> (34%).  
> 
> The engagement gains on about:home <> appear to be spread across the board:  Top Sites (from 20% to 28%, a +40% gain), Search (from 11% to 17%, a +37% gain), Highlights (from 1.6% to 2.6%, a +62% gain) and Pocket (from 0.13% to 0.34%, a 161% gain).  The explanation for this difference could be that people use the Newtab page mostly to go somewhere they already have in mind, whereas they use the Home page to discover new things or to go to places they have been to before.  
> 
> This is just one of the insights we have gained since launch.  We are using results like this to help improve the browsing experience for our users.
> 
> Performance / Perceived Performance
> 
> In the lead up to Firefox Quantum, we worked closely with the Quantum Flow team to understand what role Activity Stream played in core performance targets <https://health.graphics/quantum>.  We were eager to make sure we weren’t regressing any existing metrics and focused hard on the specific case of improving/optimizing browser startup <https://github.com/mozilla/activity-stream/issues?utf8=%E2%9C%93&q=label:Startup+>.  
> 
> In the 58 cycle, we have had a lot of conversations with Dominik and Florian about the study of the ‘perceived’ performance of systems like Activity Stream.  We identified the key events in the starting of Activity Stream and set to work improving the important browser start-up metric.  We found video A/B comparison tools to be invaluable for this work.
> 
> The link below is a video comparing Browser Startup of Firefox 57 and 58.  Note that to focus on just looking at Activity Stream, we do not include the entire startup sequence of the browser.  We have synchronized the two videos to start at the ‘first paint’ event of the overall Firefox startup sequence.
> 
> <image.png>
>  <https://drive.google.com/a/mozilla.com/file/d/1-ZyNfr9ib82s_1gMLUOYFCe6WSmU-ot8/view?usp=sharing>
> 
> 
> 
> That's pretty cool.  Notice how we 'get something on screen' early, which is a big perceived win, but also crushes the overall perf numbers from the previous version.  
> 
> We are going to keep this work up.  Our #1 priority over the next few releases is improved performance and stability.
> 
> cheers,
> tim
> 
> _______________________________________________
> firefox-dev mailing list
> firefox-dev at mozilla.org <mailto:firefox-dev at mozilla.org>
> https://mail.mozilla.org/listinfo/firefox-dev <https://mail.mozilla.org/listinfo/firefox-dev>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20171205/e6c99b90/attachment.html>


More information about the firefox-dev mailing list