Predicting trouble before it reaches users

Scaling a system the size of GitHub's means constantly balancing new features against the risk of performance degradation. Even small changes can ripple through a deeply interconnected stack. The approach that works best is a combination of deep observability, careful experimentation, and a willingness to simplify code paths that no longer justify their complexity.

Several tools form the backbone of this workflow. Metrics for every significant event flow into Datadog, where patterns can be tracked over time and sliced across dimensions to pinpoint trouble spots. Full event context goes to Splunk for deeper forensic work on specific incidents. For database performance, custom monitors flag slow and timed-out queries in MySQL before they become systemic. When a change is proposed, the Scientist library runs candidate code against the current implementation in production, comparing results and performance. Rollouts themselves are gated through Flipper feature flags, which allow incremental exposure from early-access users to a growing percentage of the population, with rollback available at any point.

Rebuilding a slow repository lookup

The first case involved a SQL query that was timing out at a high rate. Splunk traced the problem to the Command Palette, which loads a list of repositories for the current user. The original logic pulled repositories in a way that, for organizations with many active repositories, generated a SQL query with a very large IN (...) clause. That clause carried a high risk of timeout.

org_repo_ids = Repository.where(owner: org).pluck(:id)
suggested_repo_ids = Contribution.where(user: viewer, repository_id: org_repo_ids).pluck(:repository_id)

The interesting twist was that the obvious fix—querying the user first, since any given user contributes to a relatively small number of repositories—hadn't been viable in similar past situations. This use case was subtly different, and the different shape of the problem opened up a new solution.

contributor_repo_ids = Contribution.where(user: viewer).pluck(:repository_id)
suggested_repo_ids = Repository.where(owner: org, id: contributor_repo_ids)

A Scientist experiment with a candidate code block confirmed the hypothesis. Datadog showed the candidate returned identical results and improved performance by 80–90%.

Encouraged, the team took a closer look at other queries this feature generated, and found two more candidates for improvement.

The first candidate eliminated a SQL query entirely by sorting results in the application rather than in the database. A new experiment showed this actually performed 40–80% worse than the control, so the candidate was discarded and the experiment ended.

The second candidate was more promising. A query filtering repositories based on the viewer's access level was iterating through the result list one item at a time, but the access check could be batched. A new experiment with a single batched query improved performance by another 20–80%.

While these experiments were being finalized, the team scanned adjacent code for similar patterns and found another filter that could benefit from the same batching approach. That final change confirmed a 30–40% performance improvement, leaving the feature in a state that satisfied developers, database administrators, and users alike.

Shedding an unnecessary access check

Not every problem needs to wait for an outage or a pile of timeouts to justify attention. The team also reviews the busiest endpoints for each product area to get ahead of degradation.

For one team, Splunk logs tagged with controller and action pairs were used to identify the top 10 endpoints. Those were then plotted in a Datadog dashboard showing request volume, average and P99 latency, and maximum latency. The busiest action turned out to be a simple redirect whose performance regularly degraded to the timeout threshold.

Datadog's APM traces, sorted by elapsed time, showed that slow requests were spending a long time on an access check that wasn't actually required before sending the redirect response. Most Rails requests generate HTML, where shared controller filters verify viewer access before rendering—but a redirect doesn't need that verification.

A Scientist experiment wasn't an option here, since Rails controller filters are configured at boot time rather than per request. But filters can be made conditional, which allowed a Flipper feature flag to control the behavior. With the flag enabled, the controller skipped the unnecessary filters for redirect requests. Ramping up via the feature flag while monitoring Datadog for performance and status, and Splunk for anomalies, confirmed the improvement. P75 and P99 latency improved, and—more importantly—maximum latency became consistent and much less likely to time out. The change was graduated, and the pattern was generalized so other controllers with similar redirects can use it.

Lessons from iterative simplification

  • Observability pays for itself. The combination of metrics and logs was what made it possible to identify and fix these problems quickly in the first place.
  • Old problems can have new solutions. A use case that looks familiar may be different enough to make an approach viable that previously wasn't.
  • Look sideways. Fixing one issue often surfaces related issues in adjacent code that are worth tackling at the same time.
  • Don't wait for timeouts. Monitoring busy endpoints proactively means fixing something when it's merely slow rather than when it's breaking.
  • Make small, controllable changes. Gradual rollouts with measurable results keep risk low and confidence high.