When Firefox Preview shipped, it was also the official launch of Glean, our new mobile product analytics & telemetry solution true to Mozillas values. This post goes into how we got there and what it’s design principles are.

Background

In the last few years, Firefox development has become increasingly data-driven. Mozilla’s larger data engineering team builds & maintains most of the technical infrastructure that makes this possible; from the Firefox telemetry code to the Firefox data platform and hosting analysis tools. While data about our products is crucial, Mozilla has a rare approach to data collection, following our privacy principles. This includes requiring data review for every new piece of data collection to ensure we are upholding our principles — even when it makes our jobs harder.

One great success story for us is having the Firefox telemetry data described in machine-readable and clearly structured form. This encourages best practices like mandatory documentation, steering towards lean data practices and enables automatic data processing — from generating tables to powering tools like our measurement dashboard or the Firefox probe dictionary.

However, we also learned lessons about what didn’t work so well. While the data types we used were flexible, they were hard to interpret. For example, we use plain numbers to store counts, generic histograms to store multiple timespan measures and allow for custom JSON submissions for uncovered use-cases. The flexibility of these data types means it takes work to understand how to use them for different use-cases & leaves room for accidental error on the instrumentation side. Furthermore, it requires manual effort in interpreting & analysing these data points. We noticed that we could benefit from introducing higher-level data types that are closer to what we want to measure — like data types for “counters” and “timing distributions”.

What about our mobile telemetry?

Another factor was that our mobile product infrastructure that was not ideally integrated yet with the Firefox telemetry infrastructure above. Different products used different analytics solutions & different versions of our own mobile telemetry code, across Android & iOS. Also, our own mobile telemetry code did not describe its metrics in machine-readable form. This meant analysis was potentially different for each product & new instrumentations were higher effort. Integrating new products into the Firefox telemetry infrastructure meant substantial manual effort.

From reviewing the situation, one main question came up: What if we could provide one consistent telemetry SDK for our mobile products, bringing the benefits of our Firefox telemetry infrastructure but without the above mentioned drawbacks?

Introducing Glean

In 2018, we looked at how we could integrate Mozilla’s mobile products better. Putting together what we learned from our existing Firefox Telemetry system, feedback from various user interviews and what we found mattered for our mobile teams, we decided to reboot our telemetry and product analytics solution for mobile. We took input from a cross-functional set of people, data science, engineering, product management, QA and others to form a complete picture of what was required.

From that, we set out to build an end-to-end solution called Glean, consisting of different pieces:

• Product-side tools — The data enters our system here through the Glean SDK, which is what products integrate and record data into. It provides mobile APIs and aims to hide away the complexities of reliable data collection.
• Services — This is where the data is stored and made available for analysis, building on our Firefox data platform.
• Data Tools — Here our users are able to look at the data, performing analysis and setting up dashboards. This goes from running SQL queries, visualizing core product analytics to data scientists digging deep into the raw data.

Our main goal was to support our typical mobile analytics & engineering use-cases efficiently, which came down to the following principles:

• Basic product analytics are collected out-of-the-box in a standardized way. A baseline of analysis is important for all our mobile applications, from counting active users to retention and session times. This is supported out-of-the-box by our SDK and works consistently across mobile products that integrate it.
• No custom code is required for adding new metrics to a product. To make our engineers more productive, the SDK keeps the amount of instrumentation code required for metrics as small as possible. Engineers only need to specify what they want to instrument, with which semantics and then record the data using the Glean SDK.
• New metrics should be available for basic analysis without additional effort. Once a released product is enabled for Glean, getting access to newly added metrics shouldn’t require a time-consuming process. Instead they should show up automatically, both for end-to-end validation and basic analysis through SQL.

To make sure that what we build is true to Mozilla’s values, encourages best practices and is sustainable to work with, we added these principles:

• Lean data practices are encouraged through SDK design choices. It’s easy to limit data collection to only what’s necessary and documentation can be generated easily, aiding both transparency & understanding for analysis.
• Use of standardized data types & registering them in machine-readable files. By having collected data described in machine-readable files, our various data tools can read them and support metrics automatically, without manual work, including schema generation, etc.
• Introduce high-level metric types, so APIs & data tools can better match the use-cases. To make the choice easier for which metric type to use, we introduced higher-level data types that offer clear and understandable semantics — for example, when you want to count something, you use the “counter” type. This also gives us opportunities to offer better tooling for the data, both on the client and for data tooling.
• Basic semantics on how the data is collected are clearly defined by the library. To make it easier to understand the general semantics of our data, the Glean SDK will define and document when which kind of data will get sent. This makes data analysis easier through consistent semantics.

One crucial design choice here was to use higher-level metric types for the collected metrics, while not supporting free-form submissions. This choice allows us to focus the Glean end-to-end solution on clearly structured, well-understood & automatable data and enables us to scale analytics capabilities more efficiently for the whole organization.

Let’s count something

So how does this work out in practice? To have a more concrete example, let’s say we want to introduce a new metric to understand how many times new tabs are opened in a browser.

In Glean, this starts from declaring that metric in a YAML file. In this case we’ll add a new “counter” metric:

browser.usage:
  tab_opened:
    type: counter
    description: Count how often a new tab is opened. …
    …

Now from here, an API is automatically generated that the product code can use to record when something happens:

import org.mozilla.yourApplication.GleanMetrics.BrowserUsage
…
override fun tabOpened() {
  BrowserUsage.tabOpened.add()
  …
}

That’s it, everything else is handled internally by the SDK — from storing the data, packaging it up correctly and sending it out.

This new metric can then be unit-tested or verified in real-time, using a web interface to confirm the data is coming in. Once the product change is live, data starts coming in and shows up in standard data sets. From there it is available to query using SQL through Redash, our generic go-to data analysis tool. Other tools can also later integrate it, like the measurement dashboard or Amplitude.

Of course there is a set of other metric types available, including events, dates & times and other typical use cases.

Want to see how this looks in code? You can take a look at the Glean Android sample app, especially the metrics.yaml file and its main activity.

What’s next?

The first version of the Glean solution went live to support the launch of Firefox Preview, with an initial SDK support for Android applications & a priority set of data tools. iOS support for the SDK is already planned for 2019, as is improved & expanded integration with different analysis tools. We are also actively considering support for desktop platforms, to make Glean a true cross-platform analytics SDK.

If you’re interested in learning more, you can check out:

• The Glean overview in our Firefox data documentation.
• The Glean SDK and it’s documentation.
• An overview of Mozilla’s data pipeline.

We’ll certainly expand on more technical details in future upcoming blog posts.

Special thanks

While this project took contributions from a lot of people, I especially want to call out Frank Bertsch (data engineering lead), Alessio Placitelli (Glean SDK lead) and Michael Droettboom (data engineer & SDK engineer). Without their substantial contributions to design & implementation, this project would not have been possible.

Introducing Glean — Telemetry for humans was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

To make new developments more visible we aim to publish a quarterly newsletter. As we skipped one, some important items from Q4 are also highlighted this time.

This year our teams are putting their main focus on:

• Making experimentation easy & powerful.
• Providing a low-latency view into product release health.
• Making it easy to work with events end-to-end.
• Addressing important user issues with our tools.

Usage improvements

Last year we started to investigate how our various tools are used by people working on Firefox in different roles. From that we started addressing some of the main issues users have.

Most centrally, the Telemetry portal is now the main entry point to our tools, documentation and other resources. When working with Firefox data you will find all the important tools linked from there.

We added the probe dictionary to make it easy to find what data we have about Firefox usage.

For STMO, our Redash instance, we deployed a major UI refresh from the upstream project.

There is new documentation on prototyping and optimizing STMO queries.

Our data documentation saw many other updates, from cookbooks on how to see your own pings and sending new pings to adding more datasets. We also added documentation on how our data pipeline works.

Enabling experimentation

For experimentation, we have focused on improving tooling. Test Tube will soon be our main experiment dashboard, replacing experiments viewer. It displays the results of multivariant experiments that are being conducted within Firefox.

We now have St. Moab as a toolkit for automatically generating experiment dashboards.

Working with event data

To make working with events easier, we improved multiple stages in the pipeline. Our documentation has an overview of the data flow.

On the Firefox side, events can now be recorded through the events API, from add-ons, and whitelisted Firefox content. From Firefox 61, all recorded events are automatically counted into scalars, to easily get summary statistics.

Event data is available for analysis in Redash in different datasets. We can now also connect more event data to Amplitude, a product analytics tool. A connection for some mobile events to Amplitude is live, for Firefox Desktop events it will be available soon.

Low-latency release health data

To enable low-latency views into release health data, we are working on improving Mission Control, which will soon replace arewestableyet.com.

It has new features that enable comparing quality measures like crashes release-over-release across channels.

Firefox Telemetry tools

For Firefox instrumentation we expanded on the event recording APIs. To make build turnaround times faster, we now support adding scalars in artifact builds and will soon extend this to events.

Following the recent Firefox data preferences changes, we adopted Telemetry to only differentiate between “release” and “prerelease” data.

This also impacted the measurement dashboard and telemetry.js users as the current approach to publishing this data from the release channel does not work anymore.

The measurement dashboard got some smaller usability improvements thanks to a group of contributors. We also prototyped a use counter dashboard for easier analysis.

Datasets & analysis tools

To power LetsEncrypt stats, we publish a public Firefox SSL usage dataset.

The following datasets are newly available in Redash or through Spark:

• client_counts_daily — This is useful for estimating user counts over a few dimensions and a long history with daily precision.
• first_shutdown_summary — A summary of the first main ping of a client’s lifetime. This accounts for clients that do not otherwise appear in main_summary.
• churn — A pre-aggregated dataset for calculating the 7-day churn for Firefox Desktop.
• retention — A pre-aggregated dataset for calculating retention for Firefox Desktop. The primary use-case is 1-day retention.

For analysis tooling we now have Databricks available. This offers instant-on-notebooks with no more waiting for clusters to spin up and supports Scala, SQL and R. If you’re interested sign up to the databricks-discuss mailing list.

We also got the probe info service into production, which scrapes the probe data in Firefox code and makes a history of it available to consumers. This is what powers the probe dictionary, but can also be used to power other data tooling.

Getting in touch

Please reach out to us with any questions or concerns.

• You can find us on IRC in #telemetry and #datapipeline.
• We are available on slack in #fx-metrics.
• The main mailing list for data topics is fx-data-dev.
• Bugs can be filed in one of these components.
• We are on Twitter as @MozTelemetry.

Cheers from

• The data engineering team (Katie Parlante), consisting of
• The Firefox Telemetry team (Georg Fritzsche)
• The Data Platform team (Mark Reid)
• The Data Tools team (Rob Miller)

Firefox Data engineering newsletter Q1 / 2018 was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Any data that we don’t know completely ahead of time can and will behave differently than what we expected. A classic example for this is user input, say a text field. If we don’t limit the length and contents, somebody will eventually enter a book lengths worth of data or try to use it to attack a system.

But the same problem extends to data from systems we control and that we might have faith in. At Mozilla we have a variety of teams and products which are deployed to a large and diverse population of users across the world. We may think we know how those products and the data they generate behave, but practically we always find surprises.

A case study

Let’s consider a hypothetical mobile application. The mobile application has a basic system to collect product data, and makes it easy to collect new string values. To make it easier and more flexible for our teams to add something, we don’t impose a hard limit on the length of the string. We have documentation on the various instrumentation options available, making it easy to choose the best for each use-case.

Now, this works great and everybody can add their instrumentation easily. Then one day a team needs data on a specific view in the application to better understand how it gets used. Specifically they need to know how long it was visible to the user, which buttons were interacted with in which order and what the screen size of the device was. This doesn’t seem to directly fit into our existing instrumentation options, but our string recording is flexible enough to accommodate different needs.

So they put that data together in a string, making it structured so it’s reasonable to use later and we start sending it out in our JSON data packages:

...
"view_data": "{\"visible_ms\": 35953, \"buttons_used\": [\"change_name\", \"confirm\", \"help\"], \"screen_size\": \"960×540\"}",
...

The change gets tested and works fine so it gets shipped. Some time later we get a report that our product dashboards are not updated. An investigation shows that the jobs to update the dashboards were timing out, due to unusually large strings being submitted. It turns out that some users click buttons in the view 100 times or more.

What’s more, a detailed review shows that the user churn rate in our dashboard started to increase slightly, but permanently, around the time the change shipped. The favored hypothesis is that the increased data size for some users leads to lower chances of the data getting uploaded.

So, what went wrong?

To be clear, this is built as a bad example. There is a whole bunch of things that could be learnt from the above example; from getting expert review to adding instrumentation options to building the data system for robustness on both ends. However, here i want to highlight how the lack of a limit for the string length propagated through the system.

No software component exists in isolation. Looking at a high-level data flow through a product analytics system, any component in this system has a set of important parameters with resulting trade-offs from our design choices. The flexibility of a component in an early stage puts fewer constraints on the data that flows through, which propagates through the system and enlarges the problem space for each component after it.

The unbound string length of the data collection system here means that we know less about the shape of data we collect, which impacts all components in the later stages. Choosing explicit limits on incoming data is critical and allows us to reason about the behavior of the whole system.

Find the right limit

Choosing a limit is important, but that doesn’t mean we should restrict our data input as much as we can. If we pick limits that are too strict, we end up blocking use-cases that are legitimate but not anticipated. For each system that we build, we have to make a design decision on the scale from most strict to arbitrary values and weigh the trade-offs.

For me, my take-away is: Have a limit. Reason about it. Document it. The right limit will come out of conversations and lessons learnt — as long as we have one.

Know your limits was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are two notable changes here to previous versions:

• There is only one control now for Firefox data submission.
• There is a new control for SHIELD studies, which i won’t detail here.

As detailed in another post, this doesn’t change anything about the type of data being collected while it makes the data controls less confusing. However, it does have some technical consequences for Telemetry that we landed in Firefox 58.

What does this mean for Telemetry?

Previously we had two different sets of data we collected:

• “opt-out” or “base” Telemetry: This was on by default on all Firefox channels.
• “opt-in” or “extended” Telemetry: This was off by default on Firefox release and on by default on prerelease channels. Turning off “opt-out” Telemetry also turned this off.

Now the behavior is more streamlined:

• There is just one control for data upload for Firefox, which is on by default.
• Individual measurements are now collected from either all users or prerelease users.

So for Firefox we now just differentiate between “release data” and “prerelease data”. (For some specific features or studies there may still be explicit opt-in notifications, but this covers all standard Firefox data collection.)

While this has broader technical impact on Firefox data collection, the changes we landed should make this work mostly seamless.

What does this mean for me?

For Firefox users, there is no significant change; the data upload setting users chose is respected as before. The opt-in for additional Telemetry data went away; instead we always collect less data on Firefox release.

For most Firefox development, nothing should change. Telemetry takes care of doing the right thing for histograms, scalars & events internally.

Local builds from current mozilla-central should default to recording all the prerelease data but not upload it, without any extra build flags. about:telemetry allows you to confirm that.

For testing locally how release Telemetry behaves, there is a hidden pref: toolkit.telemetry.testing.overridePreRelease.

For Fennec development, nothing changed. Fennec still has the same Telemetry behavior and preference handling as before.

For writing Firefox tests, nsITelemetry.canRecordExtended can be used to check whether prerelease data is recorded. The pref toolkit.telemetry.testing.overridePreRelease can be used for tests that need to enable prerelease behavior.

For QAing Firefox data collection, you might need to override the prerelease status using the pref toolkit.telemetry.testing.overridePreRelease. The prerelease and sending status can be checked in about:telemetry.

If you are looking at Telemetry data, there are some changes. In Firefox 58, we will not receive the “extended” data from release anymore. This directly affects some parts of telemetry.mozilla.org that provided release data from that small “opt-in” population:

• the distribution and evolution dashboards will soon not show release data anymore.
• telemetry.js and the aggregates service will not provide release data when 58 is released.

If you will be affected by these changes and need alternatives, please reach out to us.

The new Telemetry behavior

Here is an overview of how the important Telemetry controls now behave for Firefox Desktop.

Currently the naming of these various controls is inconsistent and confusing. While we focused on the minimal required changes first, we will clean this up in the near future. Updating the Telemetry documentation to match the new behaviors is on-going.

First, there are the main Firefox Telemetry preferences:

datareporting.healthreport.uploadEnabled: This is our main preference that controls whether Firefox can send data and follows the user choice. It didn’t see any change in behavior from these changes.

toolkit.telemetry.enabled: This preference is now locked to reflect prerelease or release builds (true and false respectively).

This previously matched the additional data control and controlled the “extended” Telemetry collection. It is also used in other places to control collecting additional data.

More data controls are on the nsITelemetry interface:

nsITelemetry.canRecordBase: This is generally true and reflects whether any Telemetry data can be recorded. Most code should never need this.

nsITelemetry.canRecordExtended: This reflects if Firefox is on a prerelease or release builds (true and false respectively).

There is also the releaseChannelCollection property in Histograms.json, Scalars.yaml & Events.yaml. These properties keep working seamlessly, with no action required. They now follow the prerelease status of the above settings.

Reaching out

• If you have any questions reach out to :gfritzsche, :chutten, :dexter, :frank or the team.
• For any specific issues, please file a bug. We are using bug 1406390 for tracking.

Data preference changes in Firefox 58 was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

As the data platform & tools team we provide core tools for using data to other teams. This spans Firefox Telemetry, data storage and analysis to some central data viewing tools. To make new developments more visible we publish a quarterly update on our work.

In the last quarter we continued focusing on decreasing data latency, supporting analytics and experimentation workflows, improving stability and building Mission Control.

Let’s go faster

To enable faster decision making, we worked on improving latency for important use-cases.

Most notable is that the main pings now arrive much faster, which power most of our main dashboards and analysis. The new rule of thumb is 2 days until 95% of the main ping data is available, from activity in the browser to being available for analysis.

In Firefox Telemetry we can now record new probes from add-ons without having to ride the trains, which greatly reduces shipping times for instrumentation. This is available first with events in 56 and scalars in 58.

The new update ping provides a lower-latency signal for when updates are staged and successfully applied. It is queryable through the telemetry_update_parquet dataset.

Similarly, the new-profile ping is a signal for new profiles and installations, which is now queryable through the telemetry_new_profile_parquet dataset.

The new first-shutdown ping helps us to better understand users that churn after the first session, by sending the first sessions data of a user immediately on Firefox shutdown.

Enabling experimentation

This year saw a lot of cross-team work on enabling experimentation workflows in Firefox. A focus was on enabling various SHIELD studies.

Here the experiments viewer saw a lot of improvements, which provides a front-end view for inspecting how various metrics perform in an experiment.

An experiments dataset is now available in Redash and Spark, which includes data for SHIELD opt-in experiments and is based on the main_summary dataset.

The experiment_aggregates dataset now includes metadata about the experiment, and its reliability and speed have improved significantly.

Other use-cases can build on the ping data from most experiments using experiment annotations, which is available within 15 minutes in the telemetry-cohort data source.

Tools for exploring data

Our data tools make it easier to access and query the data we have. Here our Redash installation at sql.telemetry.mozilla.org saw many improvements including:

• Query revision control and reversion.
• Better security and usability for templated queries.
• Schema browser and autocomplete usability and performance improvements.
• Better support for Athena data sources.

Mission Control is a new tool, which makes key measures of a Firefox release, like crash counts, available with low latency. An early version of it is now available here.

On the Firefox side, about:telemetry got a major redesign, which makes it more easy to navigate, added a global search and aligns it with the photon design.

Powering data analysis

To make analysis more effective, two new datasets were added:

• clients_daily, which summarizes main ping data into one row per client and day.
• heavy_users (docs), which has a similar format, but contains only clients that match our definition of “heavy users”.

For analysis jobs run through ATMO, the reliability was greatly improved, which resulted in a big decrease of job failures.

Also, support was added for using of different EMR versions with different ATMO installations, allowing us to test changes to our EMR configuration much more thoroughly prior to deployment.

What is up next?

Some of the things that we will work on in the next months include:

• Firefox 56 saw data preference changes in the UI, we will follow up to align some Telemetry behavior.
• Databricks is being actively evaluated, with the goal of improving analysis productivity and reliability.
• Further usability improvements to current Experiments Viewer, and significant work done on a ground-up rewrite.
• Providing a dataset for “one day retention” analysis.
• A generic HTTP endpoint, moz_ingest, will be available to accept non-telemetry data. Data can be posted in any format but if it is JSON it can automatically tie into our schema validation capabilities.
• Collaboration with the Activity Stream team on bringing our event pipelines together.
• Activity stream is cross-checking & augmenting their experiment pipeline’s results with the experiments_summary dataset.

Get in touch

Please reach out to us with any questions or concerns.

• You can find us on IRC in #telemetry and #datapipeline.
• We are available on slack in #fx-metrics.
• The main mailing list for data topics is fx-data-dev.
• Bugs can be filed in one of these components.

You can also find us on Twitter as @MozTelemetry.

Firefox data platform & tools update, Q3 2017 was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

They are well defined and allow teams to autonomously add new instrumentation. As they are listed in machine-readable files, our data pipeline can support them automatically and new probes just start showing up in different tools. A definition like this enables views like this.

This works great when shipping probes in the Firefox core code, going through our normal release and testing channels, which takes a few weeks.

Going faster

However, often we want to ship code faster using add-ons: this may mean running experiments through Test Pilot and SHIELD or deploying Firefox features through system add-ons.

When adding new instrumentation in add-ons, there are two options:

• Instrumenting the code in Firefox core code, then waiting a few weeks until it is in release.
• Implementing a custom ping and submitting it through Telemetry, requiring additional client and pipeline work.

Neither are satisfactory; there is significant manual effort for running simple experiments and adding features.

Filling the gap

This is one of the main pain-points coming up for adding new data collection, so over the last months we were planning how to solve this.

As the scope of an end-to-end solution is rather large, we are currently focused on getting the support built into Firefox first. This can enable some use-cases right away. We can then later add better and automated integration in our data pipeline and tooling.

The basic idea is to use the existing Telemetry APIs and seamlessly allow them to record data from new probes as well. To enable this, we will extend the API with registration of new probes from add-ons at runtime.

The recorded data will be submitted with the main ping, but in a separate bucket to tell them apart.

What we have now

We now support add-on registration of events from Firefox 56 on. We expect event recording to mostly be used with experiments, so it made sense to start here.

With this new addition, events can be registered at runtime by Mozilla add-ons instead of using a registry file like Events.yaml.

When starting, add-ons call nsITelemetry.registerEvents() with information on the events they want to record:

Services.telemetry.registerEvents(“myAddon.ui”, {
 “click”: {
   methods: [“click”],
   objects: [“redButton”, “blueButton”],
 }
});

Now, events can be recorded using the normal Telemetry API:

Services.telemetry.recordEvent(“myAddon.ui”, “click”,
                               “redButton”);

This event will be submitted with the next main ping in the “dynamic” process section. We can inspect them through about:telemetry:

On the pipeline side, the events are available in the events table in Redash. Custom analysis can access them in the main pings under payload/processes/dynamic/events.

The larger plan

As mentioned, this is the first step of a larger project that consists of multiple high-level pieces. Not all of them are feasible in the short-term, so we intend to work towards them iteratively.

The main driving goals here are:

1. Make it easy to submit new Telemetry probes from Mozilla add-ons.
2. New Telemetry probes from add-ons are easily accessible, with minimal manual work.
3. Uphold our standards for data quality and data review.
4. Add-on probes should be discoverable from one central place.

This larger project then breaks down into roughly these main pieces:

Phase 1: Client work.

This is currently happening in Q3 & Q4 2017. We are focusing on adding & extending Firefox Telemetry APIs to register & record new probes.

Events are supported in Firefox 56, scalars will follow in 57 or 58, then histograms on a later train. The add-on probe data is sent out with the main ping.

Phase 2: Add-on tooling work.

To enable pipeline automation and data documentation, we want to define a variant of the standard registry formats (like Scalars.yaml). By providing utilities we can make it easier for add-on authors to integrate them.

Phase 3: Pipeline work.

We want to pull the probe registry information from add-ons together in one place, then make it available publically. This will enable automation of data jobs, data discovery and other use-cases. From there we can work on integrating this data into our main datasets and tools.

The later phases are not set in stone yet, so please reach out if you see gaps or overlap with other projects.

Questions?

As always, if you want to reach out or have questions:

• You can find us on IRC in #telemetry and #datapipeline.
• We are available on slack in #fx-metrics.
• The main mailing list for data topics is fx-data-dev.
• Bugs can be filed in one of these components.
• You can also find us on Twitter as @MozTelemetry.

Recording new Telemetry from add-ons was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

The data platform and tools teams are working on our core Telemetry system, the data pipeline, providing core datasets and maintaining some central data viewing tools.

To make new work more visible, we provide quarterly updates.

What’s new in the last few months?

A lot of work in the last months was on reducing latency, supporting experimentation and providing a more reliable experience of the data platform.

On the data collection side, we have significantly improved reporting latency from Firefox 55, with preliminary results from Beta showing we receive 95% of the “main” ping within 8 hours (compared to previously over 90 hours). Curious for more detail? #1 and #2 should have you covered.

We also added a “new-profile” ping, which gives a clear and timely signal for new clients.

There is a new API to record active experiments in Firefox. This allows annotating experiments or interesting populations in a standard way.

The record_in_processes field is now required for all histograms. This removes ambiguity about which process they are recorded in.

The data documentation moved to a new home: docs.telemetry.mozilla.org. Are there gaps in the documentation you want to see filled? Let us know by filing a bug.

For datasets, we added telemetry_new_profile_parquet, which makes the data from the “new-profile” ping available.

Additionally, the main_summary dataset now includes all scalars and uses a whitelist for histograms, making it easy to add them. Important fields like active_ticks and Quantum release criteria were also added and backfilled.

For custom analysis on ATMO, cluster lifetimes can now be extended self-serve in the UI. The stability of scheduled job stability also saw major improvements.

There were first steps towards supporting Zeppelin notebooks better; they can now be rendered as Markdown in Python.

The data tools work is focused on making our data available in a more accessible way. Here, our main tool Redash saw multiple improvements.

Large queries should no longer show the slow script dialog and scheduled queries can now have an expiration date. Finally, a new Athena data source was introduced, which contains a subset of our Telemetry-based derived datasets. This brings huge performance and stability improvements over Presto.

What is up next?

For the next few months, interesting projects in the pipeline include:

• The experiments viewer & pipeline, which will make it much easier to run pref-flipping experiments in Firefox.
• Recording new probes from add-ons into the main ping (events, scalars, histograms).
• We are working on defining and monitoring basic guarantees for the Telemetry client data (like reporting latency ranges).
• A re-design of about:telemetry is currently on-going, with more improvements on the way.
• A first version of Mission Control will be available, a tool for more real-time release monitoring.
• Analyzing the results of the Telemetry survey, (thanks everyone!) to inform our planning.
• Extending the main_summary dataset to include all histograms.
• Adding a pre-release longitudinal dataset, which will include all measures on those channels.
• Looking into additional options to decrease the Firefox data reporting latency.

How to contact us.

Please reach out to us with any questions or concerns.

• You can find us on IRC in #telemetry and #datapipeline.
• We are available on slack in #fx-metrics.
• The main mailing list for data topics is fx-data-dev.
• Bugs can be filed in one of these components.
• You can also find us on Twitter as @MozTelemetry.

Firefox data platform & tools update, Q2 2017 was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

To make new work more visible, we provide quarterly updates.

What’s new in the last few months?

On the data collection side, scalars are now supported through the pipeline, so new flag and count histograms are now disallowed on Desktop in favour of boolean and uint scalars.

Event Telemetry is now ready for first adoption. A general events table is available, a sync events table coming up and further uses are being looked at.

For documentation, we re-worked the guide for adding new Telemetry and extended the detailed data collection documentation. The prototype for making probe history more discoverable now has daily updates and supports Nightly too.

For filing or finding bugs, there is now a new Data Platform and Tools product. Note that client-side bugs still go into the separate Toolkit::Telemetry component.

The data pipeline work powers results for re:dash and custom analysis among other things. Notable recent work here includes:

• Providing efficient lookup of client histories using Hbase.
• Experimental support for Zeppelin, a new notebook type that improves Jupyter.
• TMO dashboard is now faster through a dedicated read replica and client-side caching.
• The Dataset API now has a select method to return a subset of fields.
• Providing a framework for testable Python ETL jobs generated from a template.
• Direct-to-parquet is in production, making easier to build datasets from incoming pings.

The data tools work powers tools that make data analysis more accessible across Mozilla. Updates here are:

• For re:dash, the UI improved to make the dashboard list more accessible.
• re:dash query issues were reduced by handling failing queries using exponential back-off.
• There is also a python re:dash client (h/t to emtwo), allowing programmatic generation of queries and dashboards.
• The distribution viewer is now live, making distributions of a set of important Firefox metrics available.
• The analysis service gained features like persistent cluster storage and the ability to extend cluster lifetimes.

What is up next?

For the next few months, interesting projects in the pipeline include:

• Work to decrease data latency, by sending the last ping of a Firefox session immediately. We will also start sending timely pings for new users and updates.
• Rebooting documentation, providing guidance as well as tying existing documentation together.
• Start supporting new data collection from add-ons in Telemetry, starting with events.

How to contact us.

Please reach out to us with any questions or concerns.

• You can find us on IRC in #telemetry and #datapipeline.
• The main mailing list for data topics is fhr-dev.
• Bugs can be filed in one of these components.
• You can also find us on Twitter as @MozTelemetry.

Firefox data platform & tools update, Q1 2017 was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.

To make new features more visible, we intend to provide quarterly updates, starting with this one.

What’s new in the last few months?

On the data collection side, we added scalars and added engagement measures on top. We now support recording histograms in child processes and added categorical histograms.

We improved documentation, starting from the Telemetry wiki page and updated onboarding. Further work happened on:
- improved client-side documentation
- an introduction on analyzing telemetry data
- a guide on choosing a dataset for analysis
- updates to examples for the longitudinal dataset
- exploring how to make Firefox data more discoverable

The data pipeline work powers results for re:dash and custom analysis among other things. Notable recent work here includes:
- the cross-sectional dataset is operational in re:dash
- socorro crash data is now available in re:dash
- the new dataset API improves querying raw ping data
- the self-serve analysis portal was relaunched with improved UX
- provided easy access to self-serve real-time analysis
- launched a knowledge repository to make reports discoverable in one place and easier to review

Coming soon

For the next few months, interesting projects in the pipeline include:
- event telemetry, which enables recording event data into Telemetry in a common format
- work to decrease data latency, allowing us to make decisions faster
- adding Telemetry support for Add-ons
- making it easy for new pings to show up in re:dash thanks to direct-to-parquet
- enabling efficient lookup of client histories using Hbase
- improved alerting for Telemetry probes
- creating a new crash summary dataset, to make it easier to analyze crash data

Contact us

Please reach out to us with any questions or concerns.

You can find us on IRC in #telemetry and #datapipeline.
The main mailing list for data topics is fhr-dev.
Bugs can be filed in one of these components.
You can also find us on Twitter as @MozTelemetry.

Firefox data platform update was originally published in Georg Fritzsche on Medium, where people are continuing the conversation by highlighting and responding to this story.