Socorro is a set of components for collecting, processing and reporting on crashes. It is used by Mozilla for tracking crashes of Firefox, B2G, Thunderbird and other projects. The production Mozilla install is public and hosted at https://crash-stats.mozilla.com/
The components which make up Socorro are:
There are two main parts to Socorro:
This requires both PostgreSQL, as well as the Collector, Monitor, Processor and Middleware and Web UI.
Individual crash reports are pulled from long-term storage using the /report/index/ page, for example: https://crash-stats.mozilla.com/report/index/ba8c248f-79ff-46b4-97b8-a33362121113
The search feature is at: https://crash-stats.mozilla.com/query
This requires PostgreSQL, Middleware and Web UI. It is triggered once per day by the “daily_matviews” cron job, covering data processed in the previous UTC day.
Every other page on https://crash-stats.mozilla.com is of this type, for example the Topcrashers report: https://crash-stats.mozilla.com/topcrasher/byversion/Firefox
Install dependencies
brew update
brew install python26 git gpp postgresql tcl-tk subversion mercurial
sudo easy_install virtualenv virtualenvwrapper
Set your PATH
export PATH=/usr/local/bin:$PATH
Initialize and run PostgreSQL
initdb -D /usr/local/pgsql/data -E utf8
postgres -D /usr/local/pgsql/data
Create a symbolic link to pgsql_socket
mkdir /var/pgsql_socket/
ln -s /private/tmp/.s.PGSQL.5432 /var/pgsql_socket/
Modify postgresql config
sudo editor /usr/local/pgsql/data/postgresql.conf
Ensure that timezone is set to UTC
timezone = 'UTC'
Restart PostgreSQL to activate config changes, if the above was changed
brew service restart postgresql
Install dependencies
sudo apt-get update
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:pitti/postgresql
sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install build-essential subversion libpq-dev python-virtualenv python-dev postgresql-9.2 postgresql-plperl-9.2 postgresql-contrib-9.2 rsync python2.6 python2.6-dev libxslt1-dev git-core mercurial
Modify postgresql config
sudo editor /etc/postgresql/9.2/main/postgresql.conf
Ensure that timezone is set to UTC
timezone = 'UTC'
Restart PostgreSQL to activate config changes, if the above was changed
sudo /usr/sbin/service postgresql restart
Install dependencies
sudo yum install postgresql-server postgresql-plperl perl-pgsql_perl5 postgresql-contrib subversion make rsync subversion gcc-c++ python-virtualenv mercurial
Initialize and enable PostgreSQL on startup
service postgresql initdb
service postgresql start
chkconfig postgresql on
Modify postgresql config
sudo vi /etc/postgresql/9.2/main/postgresql.conf
Ensure that timezone is set to UTC
timezone = 'UTC'
Restart PostgreSQL to activate config changes, if the above was changed
sudo /usr/sbin/service postgresql restart
Create a superuser account for yourself, and the breakpad_rw account for Socorro to use
sudo su - postgres -c "createuser -s $USER"
psql -c "CREATE USER breakpad_rw" template1
psql -c "ALTER USER breakpad_rw WITH ENCRYPTED PASSWORD 'aPassword'" template1
Clone from github
git clone https://github.com/mozilla/socorro
By default, you will be tracking the latest development release. If you would like to use a stable release, determine latest release tag from our release tracking wiki: https://wiki.mozilla.org/Socorro:Releases#Previous_Releases
git checkout $LATEST_RELEASE_TAG
Copy the .ini-dist files in config/ as necessary. The rest of this guide will assume that the defaults are used.
Clone from github
git clone https://github.com/mozilla/socorro-crashstats
Read the INSTALL.md for installation instructions.
By default, you will be tracking the latest development release. If you would like to use a stable release, determine latest release tag from our release tracking wiki: https://wiki.mozilla.org/Socorro:Releases#Previous_Releases
git checkout $LATEST_RELEASE_TAG
This is the binary which processes breakpad crash dumps into stack traces:
make minidump_stackwalk
To run and hack on Socorro apps, you will need:
1) all dependencies installed from requirements/{prod,dev}.txt 1.1) also requirements/dev.txt, if you are doing development 2) set PYTHONPATH to .
Socorro can install the dependencies into a virtualenv for you, then just activate it and set your PYTHONPATH
make virtualenv
. socorro-virtualenv/bin/activate
export PYTHONPATH=.
Or you can choose to manage the virtualenv yourself, perhaps using virtualenwrapper or similar.
Before loading the schema, make sure to load the roles required for Socorro. You should edit change the passwords, for a production install.
psql -f sql/roles.sql
Load the Socorro schema
./socorro/external/postgresql/setupdb_app.py --database_name=breakpad
IMPORTANT NOTE - many reports use the reports_clean_done() stored procedure to check that reports exist for the last UTC hour of the day being processed, as a way to catch problems. If your crash volume does not guarantee one crash per hour, you may want to modify this function in socorro/external/postgresql/raw_sql/procs/reports_clean_done.sql and reload the schema
./socorro/external/postgresql/setupdb_app.py --database_name=breakpad --dropdb --database_superuser=your_superuser --database_superuserpassword=bPassword
By default, setupdb_app.py will use ‘breakpad_superuser’ as the superuser, and ‘bPassword’ as the password. This is required because ‘breakpad_rw’ user must not be a superuser in the database.
If you want to hack on Socorro, or just see what a functional system looks like, you also have the option to generate and populate the DB with synthetic test data
./socorro/external/postgresql/setupdb_app.py --database_name=breakpad --fakedata --dropdb --database_superuser=your_superuser --database_superuserpassword=bPassword
Socorro uses PostgreSQL partitions for the reports table, which must be created on a weekly basis.
Normally this is handled automatically by the cronjob scheduler crontabber but can be run as a one-off:
python socorro/cron/crontabber.py --job=weekly-reports-partitions --force
Copy default config files
cp config/collector.ini-dist config/collector.ini
cp config/processor.ini-dist config/processor.ini
cp config/monitor.ini-dist config/monitor.ini
cp config/middleware.ini-dist config/middleware.ini
Run Socorro servers - NOTE you should use different terminals for each, perhaps in a screen session
python socorro/collector/collector_app.py --admin.conf=./config/collector.ini
python socorro/processor/processor_app.py --admin.conf=./config/processor.ini
python socorro/monitor/monitor_app.py --admin.conf=./config/monitor.ini
python socorro/middleware/middleware_app.py --admin.conf=config/middleware.ini
If you want to modify something that is common across config files like PostgreSQL username/hostname/etc, make sure to see config/common_database.ini-dist and the “+include” line in the service-specific config files (such as collector.ini, processor.ini and monitor.ini). This is optional but recommended.
Configure socorro-crashstats/crashstats/settings/local.py to point at your local middleware server
MWARE_BASE_URL=http://localhost:8883
Refer to Production Install for information about installing Socorro for production use.
Generate a test crash:
See: https://developer.mozilla.org/en/Environment_variables_affecting_crash_reporting
If you already have a crash available and wish to submit it, you can use the standalone submitter tool (assuming the JSON and dump files for your crash are in the ”./crashes” directory)
python socorro/collector/submitter_app.py -u http://crash-reports/submit -s ./crashes/
You should get a “CrashID” returned. Check syslog logs for user.*, should see the CrashID returned being collected.
Attempt to pull up the newly inserted crash: http://crash-stats/report/index/YOUR_CRASH_ID_GOES_HERE
The (syslog “user” facility) logs should show this new crash being inserted for priority processing, and it should be available shortly thereafter.