Glossary¶

breakpad¶

Breakpad is a set of client and server components that implement a crash reporting system.

Mozilla currently uses a heavily patched Breakpad library.

Mozilla is working on replacing Breakpad and other tools with new ones written in Rust.

See also

Breakpad: Breakpad project code and documentation.

Mozilla’s patched Breakpad

crash annotation¶

A crash annotation is a key/value pair with some bit of information.

Crash annotations are one of two kinds of data in a Breakpad-style crash report the other being minidumps.

In a crash report, all values are strings, but they can be decoded as a variety of data types like strings, integers, floats, and JSON-encoded structures.

Example:

ProductName=Firefox

Example:

Version=115.3.0

Example:

BuildID=20231004091611

Example:

CPUMicrocodeVersion=0xb4

Example (wrapped):

MozCrashReason=Shutdown hanging at step AppShutdownConfirmed. Something is blocking th
e main-thread.

See also

Crash Annotations

CrashAnnotations.yaml: Available crash annotations.
Socorro crash annotations schema: Socorro schema for supported crash annotations including documentation, access controls, links to data reviews, and links to relevant bugs. This information can be seen in the Crash Reporting Data Dictionary.

crash id¶

uuid¶

ooid¶

A unique identifier for the crash report. The collector generates the crash id and returns it to the crash reporter in the HTTP response. The crash reporter may keep it so that users have a list of crash reports they submitted with links to the the crash report on Crash Stats.

Crash ids look like this:

de1bb258-cbbf-4589-a673-34f800160918
                             ^^^^^^^
                             ||____|
                             |  yymmdd
                             |
                             throttle result

Historically, Antenna used the throttle result to encode the throttle result so that we knew which crash reports were collected and accepted for processing and which ones were accepted and stored, but not accepted for processing. Antenna no longer accepts crash reports for storing, but not for processing, so this character is now always 0.

Because of this structure, you can look at the last 6 characters and know when the crash report was collected.

The crash id is also referred to as “uuid”. The collector stores the crash id in the “uuid” key in the raw crash.

(Deprecated) The crash id also used to be referred to as “ooid”, but that term is deprecated and we should remove its usage everywhere.

See also

Collector-added fields: List of fields the collector adds when accepting a crash report which includes “uuid”.
Antenna spec v1 (2017): Original specification for the Antenna rewrite of the Socorro collector which includes a section on crash id structure.

crash report¶

A crash report consists of crash annotations and minidumps. It is the data packet that is sent by the crash reporter, accepted by the collector, and processed by the processor.

Socorro doesn’t accept all incoming crash reports. The collector has a throttler which rejects some crash reports.

Crash reports get rejected for a variety of reasons:

The crash report is malformed in some fundamental way.
The crash report is from a cohort we get many crash reports from and we don’t need them all. For example, we only accept 10% of Firefox desktop, release channel, Windows crash reports.
The crash report contains something we cannot accept. There have been bugs in the crash reporter in the past where it sent crash reports the user did not consent to. We reject crash reports that have the markers of these bugs.

See also

Specification: Submitting crash reports: Specification covering the structure of a crash report.

crash reporter¶

When a product crashes, the crash reporter kicks in, captures information about the state of the crashed process, product, and system, and assembles a crash report.

For Breakpad-style crash reporters, the crash report consists of crash annotations and zero or more minidumps.

See Specification: Submitting crash reports for the structure of a crash report and how it’s submitted.

See also

Firefox crash reporter: Code for the Firefox crash reporter.
Fenix crash reporter: Code for the Fenix crash reporter.

crash signature¶

The Socorro processor generates a crash signature for every crash report.

Crash signatures help us group similar crashes glossing over differences in operating system versions, platforms, architectures, drivers, video cards, web sites, etc.

Roughly, a signature consists of some flags followed by “interesting” symbols from the stack.

Examples of signatures:

OOM | small

shutdownhang | nsThreadManager::SpinEventLoopUntilInternal

mozilla::dom::ServiceWorkerRegistrar::GetShutdownPhase

<style::stylesheets::rules_iterator::RulesIterator<C> as core::iter::traits::iterator::Iterator>::next

See also

Signature Generation: Documentation on Socorro’s signature generation.

dump¶

minidump¶

A minidump is a file format for storing information about a crashed process. It contains CPU information, register contents, stacks for the crashed thread and other threads, some interesting parts of the heap, list of loaded modules, list of unloaded modules, etc.

Minidumps are smaller than core dumps which makes them handy for crash reporting.

Minidumps are created and manipulated using the Breakpad library and the rust-minidump tools.

See also

Minidump Files (Microsoft): Documentation on minidump file format.
Breakpad: minidump file format: Breakpad documentation on minidump file format.
Breakpad: processing minidumps: Breakpad documentation on processing minidumps.
rust-minidump: Type definitions, parsing, and analysis for the minidump file format.

processed crash¶

The Socorro processor takes crash annotations and minidumps, runs them through the processing pipeline, and generates a processed crash.

The processed crash contains normalized and validated data derived from the original crash report.

See also

Socorro processed crash schema: Socorro processed crash schema including descriptions, access controls, source annotations (when appropriate), and other things. This information can be seen in the Crash Reporting Data Dictionary.

protected data¶

Socorro categorizes crash report data in two ways:

public data
protected data

Public data is anything in Category 1 (technical data) and Category 2 (interaction data).

Protected data is anything more sensitive than that.

By default, all data in the raw and processed crash is considered protected. In order for it to be marked as public, we require a data review and for it to be explicitly marked as public in the relevant schema.

See also

Socorro protected data access policy: Socorro’s protected data access policy which covers who is allowed access to the data, what you can do with it, and how to get protected data access.
Data collection categories: Definitions of data collection categories.

raw crash¶

The Socorro collector parses the HTTP POST payload into a set of crash annotations and minidumps. It collects the crash annotations along with some metadata generated at collection in a raw crash structure. It saves this to AWS S3.

The Socorro processor takes the raw crash and minidumps and passes them through the processing pipeline to generate a processed crash.

The collector tries to save the crash annotation data in the raw crash as it received it. There are some exceptions:

Some crash annotations are no longer allowed to be collected. The collector will drop these before creating the raw crash. For example, we no longer collect the Email crash annotation.
Annotations that raise some kind of parse error are dropped.

When this happens, a note will be added to the “collector notes” which can be seen in the Debug tab of the report view in Crash Stats.

See also

Socorro crash annotations schema: Socorro schema for supported crash annotations including documentation, access controls, links to data reviews, and links to relevant bugs. This information can be seen in the Crash Reporting Data Dictionary.

reprocess¶

Socorro can reprocess crash reports. Reprocessing involves starting with the original crash data and running it through the processing pipeline again.

Sometimes we reprocess crash reports after we’ve made changes to signature generation so that the crash reports pick up new crash signatures.

Sometimes we reprocess crash reports after we’ve uploaded missing symbols files so minidump processing has symbols files to work with. This results in improved symbolicated stacks and new crash signatures.

Sometimes we reprocess crash reports that were affected by a bug we’ve just fixed in the processor.

stackwalker¶

A command-line minidump processor used by the Socorro processor to parse a minidump and generate a JSON-encoded digest of the minidump with symbolicated stacks, modules, hardware information, and other things.

See also

rust-minidump: Type definitions, parsing, and analysis for the minidump file format.
rust-minidump stackwalker JSON schema: Schema for the stackwalker output.