gecko.readthedocs.org "outage" and future plans

Gregory Szorc gps at mozilla.com
Tue Aug 22 22:52:21 UTC 2017


tl;dr https://gecko.readthedocs.org broke hard a few hours ago (but the
docs are still available at a temporary URL) and we have plans to move away
from Read the Docs due to ongoing reliability and control concerns.

The Firefox source repository has in-tree Sphinx docs. We've been hosting
those on https://gecko.readthedocs.org/ for a few years.

In recent months, the reliability of Read The Docs (RTD) for generating the
Firefox docs has deteriorated slowly. ~50% of docs "builds" fail for
"intermittent" reasons (mostly related to the scale of our docs AFAICT).

A few hours ago, failure to generate the docs has somehow resulted in RTD
showing a generic template page and the old docs are no longer visible. And
instead of a useful and potentially actionable build log, it shows "An
unexpected error occurred."

Because of the deterioration of RTD's reliability, the unplanned fire
drills that changes to RTD inflict upon us, and the inability to fully
control the docs generation environment to do advanced things, we've been
wanting to move away from gecko.readthedocs.org for several months. (For
the record, I have engaged RTD about reliability issues in the past. It is
clear Firefox's docs are a 0.1% use case in terms of scale and therefore
difficult to support.)

We now have TaskCluster automation uploading the generated docs whenever
they are changed on mozilla-central (bug 1382729). This is covered by
Firefox's automation "SLA" which means people notice quickly when it breaks
(unlike RTD). The next step is bug 1392765 to get a new hostname and x509
certificate so there is a more official-looking home. From there, we'll
likely do some things around hosting different versions corresponding to
Firefox releases (bug 1390693) and auto-generated documentation for
JavaScript files courtesy of sphinx-js (bug 1389341). And, yes, we'll
likely try to redirect gecko.readthedocs.org to the new home so old links
still work.

I'll send out an announcement when we have a new formal home for the
in-tree docs (hopefully in a few days). In the mean time, if you must
access the docs that were formerly available at
https://gecko.readthedocs.org/, you can find them at
http://gecko-docs.mozilla.org.s3.amazonaws.com/index.html. That's a
temporary home, so please don't link to it from anything official.

I apologize for any inconvenience this has caused. The service reliability
problems should go away once we complete the transition from RTD to
TaskCluster.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/firefox-dev/attachments/20170822/f8f05c48/attachment.html>


More information about the firefox-dev mailing list