Why Python apps still need native crash reporting

Dropbox’s desktop client is partly written in Python, a language that shields developers from many of the memory-safety pitfalls of lower-level code. But “safe” doesn’t mean “unbreakable.” A significant share of crashes originate outside Python: in the interpreter itself, in C/C++ extensions, or in platform-specific integration code such as COM on Windows and Objective-C on macOS. As the application has grown more complex, so has the share of non-Python code—and with it, the risk of dangling pointers, data races, and unchecked array accesses. A single stack trace can now span Python, C++, Objective-C, and C frames.

The limits of in-process signal handling

For years, Dropbox relied on a signal handler running inside the main process. On a fatal signal such as SIGFPE, it would capture the Python stack trace for each thread via the faulthandler module, grab the native stack trace using libc's backtrace and backtrace_symbols, then attempt to upload the data. This approach worked, but suffered from two fundamental weaknesses:

  • Blind spots at startup. If the failure occurred before the handler was installed—an ImportError, a missing library, or an installation error—nothing was captured. These “boot errors” are the most severe since they prevent the application from launching at all. Engineers only learned about them via customer support tickets, which made startup code a risky area to modify.
  • A fragile handler. The handler both captured state and transmitted it. It frequently generated a report but failed to send it. Worse, if the interpreter state itself was corrupted, the handler could not reliably extract the Python stack trace—or could derail the entire handling process.

Signal handling in Python introduces another layer of complexity. Python’s signal module only allows handlers to run on the main thread, and not synchronously. That asynchronicity meant some of the most common SIGSEGVs could not be trapped from Python in the first place.

Moving the reporter out of process with Crashpad

The solution was to move crash capture outside the application process entirely. Both Windows and macOS provide native facilities for trapping out-of-process crashes, and the Chromium project’s Crashpad library packages this functionality for standalone use.

Crashpad runs as a small helper process that monitors the application and, when a crash occurs, collects:

  1. The reason for the crash and the originating thread
  2. Stack traces for all threads
  3. Contents of portions of the heap
  4. Extra developer-defined annotations

This data is packaged into a minidump, a Microsoft-authored format originally developed for Windows and similar to a Unix core dump. The format is openly documented, and mature server-side tooling—primarily from Google and Mozilla—exists to process it.

The Crashpad architecture uses an in-process “client” object that signals an out-of-process “handler” when it detects a crash. Choosing Crashpad was straightforward given its track record in Chromium, one of the most widely deployed desktop applications. It also offered far better Windows support than the signal-based faulthandler: Crashpad leverages Windows Structured Exception Handling (SEH) to catch a broader range of fatal exceptions that signals—a UNIX concept—cannot reach.

Linux support in Crashpad arrived only recently, after Dropbox’s initial deployment. On Linux, where Crashpad was unavailable at the time, the in-process signal handler remains in use, but that will be revisited.

Mapping crashes back to source

Production builds of Dropbox ship stripped of symbols and with optimizations enabled. To make minidumps meaningful, the crash pipeline maps stack traces back to source via “symbolication.” Symbols for every build are preserved on internal servers; symbol generation is treated as part of the build itself—a failure to generate symbols blocks the release.

When a minidump arrives, the symbols for that specific build are used to resolve each stack frame, with platform-specific symbols filling in the gaps for system libraries. Microsoft provides public symbol servers covering all Windows builds. Apple does not offer an equivalent, so Dropbox caches symbols from macOS frameworks across a wide range of OS versions using test VMs, accepting occasional gaps in coverage.

Watchdog: a sidecar for exit monitoring

Rolling out new crash reporting to millions of installations required validation that the new mechanism actually worked—and that it was catching real crashes. Not all terminations are crashes; users quit applications, and updates restart them. Still, unexpected exits can be symptoms of deeper problems. The team wanted a way to record and classify exits alongside crashes, and to prove that Crashpad captured a high proportion of actual crashes.

That led to a second companion process, named “watchdog.” Deliberately simple for maximum reliability, watchdog has one job: when the desktop app exits, it captures the exit status and determines whether the shutdown was graceful (user- or app-initiated) or forced. A start event is recorded on application launch, giving the team a basis for comparing start-and-exit pairs. This measures monitoring accuracy itself—firewalls, corporate policies, and other software prevent 100% coverage—and lets the team cross-check exit codes against Crashpad reports: if an exit code implies a crash, the corresponding crash report should exist for the vast majority of users.

The watchdog allows us to validate that our crash reporting is successful
The watchdog allows us to classify crashes and terminations in a single graph

Watchdog was written in Rust for three reasons:

  1. The language’s safety guarantees make the code easier to trust.
  2. Its operating-system abstractions are well designed and part of the standard library, with straightforward FFI extension when needed.
  3. Dropbox already had significant Rust expertise, smoothing the project’s ramp-up.

From Native Faults to Python Traces

Crashpad's design assumes native code, which is reasonable given its origins in Chromium. But the Dropbox desktop client is mostly Python. A native crash report from an interpreter process that lacks Python context produces a stack trace like this:

0  _ctypes.cpython-35m-darwin.so!_i_get + 0x4
 1  _ctypes.cpython-35m-darwin.so!_Simple_repr + 0x4a
 2  libdropbox_python.3.5.dylib!_PyObject_Str + 0x8e
 3  libdropbox_python.3.5.dylib!_PyFile_WriteObject + 0x79
 4  libdropbox_python.3.5.dylib!_builtin_print + 0x1dc
 5  libdropbox_python.3.5.dylib!_PyCFunction_Call + 0x7a
 6  libdropbox_python.3.5.dylib!_PyEval_EvalFrameEx + 0x5f12
 7  libdropbox_python.3.5.dylib!_fast_function + 0x19d
 8  libdropbox_python.3.5.dylib!_PyEval_EvalFrameEx + 0x5770
 9  libdropbox_python.3.5.dylib!__PyEval_EvalCodeWithName + 0xc9e
10  libdropbox_python.3.5.dylib!_PyEval_EvalCodeEx + 0x24
11  libdropbox_python.3.5.dylib!_function_call + 0x16f
12  libdropbox_python.3.5.dylib!_PyObject_Call + 0x65
13  libdropbox_python.3.5.dylib!_PyEval_EvalFrameEx + 0x666a
14  libdropbox_python.3.5.dylib!__PyEval_EvalCodeWithName + 0xc9e
15  libdropbox_python.3.5.dylib!_PyEval_EvalCodeEx + 0x24
16  libdropbox_python.3.5.dylib!_function_call + 0x16f
17  libdropbox_python.3.5.dylib!_PyObject_Call + 0x65
18  libdropbox_python.3.5.dylib!_PyEval_EvalFrameEx + 0x666a
19  libdropbox_python.3.5.dylib!__PyEval_EvalCodeWithName + 0xc9e
20  libdropbox_python.3.5.dylib!_PyEval_EvalCodeEx + 0x24
21  libdropbox_python.3.5.dylib!_function_call + 0x16f
22  libdropbox_python.3.5.dylib!_PyObject_Call + 0x65
... on and on

That trace gives a developer little to work with. Python's faulthandler captures Python frames by executing inside the crashing process, but Crashpad is an out-of-process handler. When the target process is suspended, Crashpad can read its memory but cannot run any code in it. The solution therefore has three parts: locate the Python runtime's data structures in memory, walk those structures to reconstruct the executing Python code, and attach that information to the crash report.

Crashpad is designed to be extended, which made this feasible. We added Python stack capture to the ProcessSnapshot class and defined a custom minidump stream to hold the data.

Finding Python's Thread State

Each native thread running Python code holds a pointer to a PyThreadState via platform-specific thread-local storage. Crashpad can read that storage from the crashed process, provided it knows which thread-local storage key Python allocated during interpreter initialization.

The key is not a fixed constant; it is determined at runtime. However, Dropbox ships a custom CPython fork, giving us control over its layout. We modified that fork to publish the interpreter's runtime state—including the thread-local storage key—in a named section of the binary, using __attribute__ on macOS/Linux and __declspec on Windows. Crashpad already uses the same mechanism for its own annotations, so reading this data was straightforward.

This approach aligns with CPython's own direction: recent versions consolidate interpreter state into a single _PyRuntime struct (in Python/pylifecycle.c), which includes the thread-local storage key. We upstreamed the change as a pull request to CPython.

On Windows, grabbing thread-local state required some extra care. We contributed fixes to Crashpad to expose the relevant offsets in the Thread Environment Block.

Walking Python Frames from a Foreign Process

Once we have the thread-local storage key, we can read the current PyThreadState for any thread. That struct holds a stack of PyFrameObject entries, each pointing to a PyCodeObject with the function name, file name, and line number.

Crashpad cannot call into Python to traverse this. Instead, it must read the raw bytes from the crashed process and interpret them as copies of the relevant Python structs. That is brittle across CPython versions, so we relied on automated tests to force a synchronized update whenever Python's core structs change.

Decoding the fields themselves has a few subtleties:

  • Function and file names are Python string objects, which sit in a type hierarchy. We assumed ASCII encoding, so they map to the simple PyASCIIObject.
  • Line numbers are not stored directly. Each PyCodeObject keeps a compressed mapping (co_lnotab) from bytecode offsets to source lines. The documented decoding algorithm had to be re-implemented in Crashpad.

During the Python 2 to Python 3 migration, we maintained separate parsers in the Crashpad fork to account for the structural differences between the two runtimes.

Reconstructing the Trace on the Server

Capturing the frames is only half the work. Native stack traces interleave Python evaluation frames (PyEval_EvalFrameEx) with C and C++ calls. To make the reports legible, our server-side crash management system, Crashdash, walks the native stack again. For each frame that calls PyEval_EvalFrameEx, it pairs that frame with the next captured PyFrameObject. The result is a unified view:

file "ui/common/tray.py", line 758, in _do_segfault
file "dropbox/client/ui/cocoa/menu.py", line 169, in menuAction_
file "dropbox/gui.py", line 274, in guarantee_message_queue
file "dropbox/gui.py", line 299, in handle_exceptions
file "PyObjCTools/AppHelper.py", line 303, in runEventLoop
file "ui/cocoa/uikit.py", line 256, in mainloop
file "ui/cocoa/uikit.py", line 929, in mainloop
file "dropbox/client/main.py", line 3263, in run
file "dropbox/client/main.py", line 6904, in main_startup
file "dropbox/client/main.py", line 7000, in main

With this in place, engineers can investigate crashes regardless of whether they originate in Python, C, C++, or Objective-C. The added monitoring we built to measure the system's reliability also gave us more confidence in the client's overall health. During the largest Python 3 migration in Dropbox's history, this crash reporting infrastructure meant we could ship the transition without betting on users being unaffected by regressions.