A rare session bug and how it happened
On March 8 we logged all GitHub.com users out after confirming a rare race condition could return one user's session cookie to another user. What follows is the technical breakdown of that incident: the conditions that triggered it, how we traced it to a multi-layered interaction between a background thread and the env object reused by our HTTP server, and the fixes we've deployed.
The first reports
On March 2, 2021, a user reported being suddenly authenticated as another user while using GitHub.com logged in as themselves. They immediately logged out and flagged the issue. A few hours later, a second user reported the same experience. Reviewing request and audit logs confirmed the behavior: a session was shared across two IP addresses around the time of the reports.
We activated our security incident response procedures and pulled in security, engineering, and support teams to investigate. Because this was new behavior, we focused initially on recent infrastructure changes.
Ruling out the obvious suspects
We had recently upgraded components in our load balancing and routing tier, including a fix for an HTTP keepalive issue that seemed potentially related. After investigating, we ruled that out: the affected user requests touched different machines in that tier, so responses were not being swapped at the protocol level.
What the logs did show was something we could use: the requests that produced the incorrect session were handled on the same machine, in the same process. Our main Ruby on Rails application runs on the Unicorn Rack web server in a multi-process setup. From the logs we learned that the response body was correct—the HTML and other content matched the authenticated user — but the session cookie was wrong. The users in the reports each received a session cookie belonging to a user whose request had recently been handled in the same process. Sometimes the two requests were sequential; sometimes other requests came in between.
That pointed to state leaking between requests inside a single Ruby process. We turned to a recently rearchitected performance improvement: previously, checking a user's enabled features happened during request handling; the new code moved that check into a background thread that refreshed on an interval. That change seemed to touch the relevant code paths, and its thread safety behavior became the focus of our investigation.
The thread safety trap
The context: our main Rails application has components not written to run safely in multiple threads. Historically, that meant an exception might report an incorrect value internally, but never a user-facing change of state. Threads were used elsewhere in the app already, but the new background thread created a novel interaction with our exception handling.
When an exception occurred in the background thread—say, a query timeout — the error log contained data from both the background thread and the currently running request. We initially treated that as an internal reporting problem: it looked like the error reporting was pulling request data across threads, but we considered it safe because each request gets its own request data and Rails instantiates a new controller for each request.
The breakthrough came when we identified that Unicorn does not create a fresh env object per request. Instead, it allocates one Ruby Hash and clears it with Hash#clear between requests. That meant the thread safety issue in exception logging could share not just logged data but actual request state.
Our reproduction attempts confirmed an additional condition: an anonymous request had to start the sequence. The full chain of events:
- In the request thread, an anonymous request (Request #1) started. It registered callbacks in the current context for our internal exception reporting library. Those callbacks referenced the current Rails controller, which had access to the single Rack environment object.
- In the background thread, an exception occurred. The exception reporter copied the current context, including the callbacks from Request #1 and their reference to the Rack environment.
- In the main thread, a signed-in request (Request #2) started.
- In the background thread, the reporter processed the callbacks. One callback reads the user session identifier, but Request #1 had no authentication, so the data wasn't yet loaded. That triggered a new authentication call through the controller from Request #1. The controller pulled the session cookie from the shared Rack environment — which at that moment contained Request #2's cookie.
- In the main thread, Request #2 finished.
- Another signed-in request (Request #3) started and completed its authentication step.
- Back in the background thread, the controller finished Request #1's authentication by writing a session cookie to the cookie jar in the Rack environment. That was now the cookie jar for Request #3.
- Request #3's response went to the user with Request #2's session cookie.
If an exception fired at just the wrong time and requests processed in just the wrong sequence across multiple requests, we replaced the session in a response with one from an earlier response. Only the cookie was wrong—never the body content. This matched the log evidence exactly. The bug required several overlapping conditions: a background thread, shared exception context, callbacks in that context, the reused env object, and our authentication system. As How Complex Systems Fail describes, it took multiple failures in sequence to produce this outcome.
The fixes
We prioritized removing two of the necessary conditions. First, we removed the new background thread introduced in the performance re-architecture. The change was easy to revert because we knew exactly what had been added. That deployment went to production on March 5, and we confirmed the required conditions could no longer be met.
Next, we created and deployed a patch for Unicorn to stop sharing the environment hash between requests. That landed on March 8, adding a second layer of isolation between requests even if thread safety issues occur in the future.
On the detection side, we examined logging data for patterns consistent with incorrect session returns and manually reviewed every matching log to determine whether two sessions had in fact been conflated. Between March 5 and March 8 that analysis showed the bug had occurred, but only rarely.
We then took the additional precaution of revoking all active GitHub.com sessions. Log analysis can confirm sessions that were swapped and used, but it cannot fully rule out a session being swapped and never used. Given the potential impact of one such session being used later, we chose not to accept that risk. Revoking all sessions closed that possibility completely.
Ongoing hardening
The Unicorn patch was upstreamed with the maintainer's collaboration. Allocating a fresh environment hash for each request removes the possibility that one request can hold a reference to an object that affects the next one—an extra guard rail for other Unicorn users facing similar thread safety issues.
Inside our own codebase, we're also removing the callbacks from our exception logging context. Those callbacks let us defer work and avoid paying an overhead when unnecessary, but they made race conditions harder to debug and could retain references to long-finished requests. Plainer exception handling is easier to reason about. We are also simplifying the code paths that manage session cookies.
Longer term, we are improving thread safety across our exception handling and instrumentation. This week's fix removes one bug; the broader goal is to build code that cannot produce this class of issue at all.
Lessons from a threaded-code failure
This incident is a reminder that threaded code is fundamentally harder to reason about than single-threaded logic, particularly in a Rails codebase that has historically avoided long-running threads. In response, we have paused the use of long-running threads in our Rails processes entirely. This gives us the space to harden the surrounding code so it is safe to run in a variety of threading contexts and to build a thread-safe architecture we can rely on going forward.
Debugging this was not solely a technical exercise. Identifying the interaction of several threads, deferred callbacks, and shared objects required deep log analysis and a review of recent changes across both code and infrastructure. That work was accelerated significantly by a mature product security incident response process. Having a collaborative team spanning support, security, and engineering allowed us to quickly triage the ambiguous reports, validate the risk, and set the right priorities across the company.
Once the root cause became apparent, we focused on shipping mitigations that limited customer impact. The confidence gained from that process was what allowed us to make the difficult call to log every user out of GitHub.com. We came away with a better understanding of how our complex systems behave under concurrency, and we are turning that understanding into safeguards intended to prevent a recurrence.
