When emergency defenses become permanent problems

GitHub recently experienced an outage of sorts — not from an attack or infrastructure failure, but from its own protective measures. Users reported receiving "too many requests" errors while browsing normally, such as following links from external apps or reading pages without any rapid-fire behavior. The root cause: rate-limiting rules that were added during past abuse incidents had been left running long after their original threat had subsided.

These rules used composite signals combining industry-standard fingerprinting with GitHub-specific business logic. At the time they were created, those patterns strongly correlated with abusive traffic. But as legitimate client behavior evolved, the same patterns began matching ordinary logged-out activity. The false-positive rate was low — only about 0.5–0.9% of traffic matching the suspicious fingerprints was blocked — but the impact was decisive: users hitting all the criteria were blocked 100% of the time.

Screenshot of a 'Too many requests' screen encountered by users.
Users encountered a “Too many requests” error during normal browsing.

Why stale mitigations are hard to spot

GitHub's defense infrastructure is multi-layered, built from custom protections layered on top of extensible open-source systems like HAProxy. Each layer has valid reasons to block or throttle requests, and during an incident a mitigation might be added at any one of them. This flexibility creates a subtle problem: when a user is blocked, determining which layer made the decision requires correlating logs across systems with different schemas.

In this case, the investigation worked backward from user reports:

  1. User reports established timestamps and behavior patterns.
  2. Edge tier logs confirmed requests reaching the infrastructure.
  3. Application tier logs showed 429 "Too Many Requests" responses.
  4. Protection rule analysis revealed which specific rules matched.
Diagram showing user requests flowing through multiple infrastructure layers (Edge, Application, Service, Backend), with protection mechanisms at each layer including DDoS protection, rate limits, authentication, and access controls.

The core lesson is that observability applies to defenses as much as features. Every layer produces logs, but cross-referencing them to trace a block decision is only as good as the tooling you've built for it. In this case, the scattered log schemas meant that a rule sitting in one layer could silently affect users for months without raising any internal alarms.

The lifecycle problem

Each rule was justified when deployed. Emergency responses during active incidents require broad controls deployed quickly, and that's the right call in the moment. But without consistent lifecycle management — expiration dates, post-incident review, ongoing impact monitoring — those temporary measures became permanent technical debt. Their side effects compounded quietly as threat patterns shifted and legitimate tools changed.

Diagram showing incident mitigation lifecycle: control added during incident, works initially, remains active over time without review, eventually blocks legitimate traffic.

The overall false-positive rate, relative to total traffic, was small. But for the users it affected, the disruption was real: normal browsing interrupted by errors that should never have applied to them.

Chart showing false positives as approximately 0.003-0.004% of total traffic, with a reference line at 100%
False positives represented roughly 0.003-0.004% of total traffic.

Immediate fix and long-term changes

GitHub reviewed the problematic mitigations, analyzing what each rule blocked at the time of the incident versus what it was blocking when users complained. Rules that no longer served their purpose were removed; protections against ongoing threats were kept.

Chart showing false positive rate over 60 minutes, hovering around 0.003-0.004%
In the hour before cleanup, approximately 3-4 requests per 100,000 (0.003-0.004%) were incorrectly blocked.

The company also outlined broader improvements to prevent recurrence:

  • Better cross-layer visibility to pinpoint where blocks and rate limits originate.
  • Default treatment of incident mitigations as temporary — making one permanent should require deliberate, documented justification.
  • Post-incident practices that evaluate emergency controls and upgrade them into sustainable, targeted solutions.
Chart showing percentage of fingerprint matches that were blocked by also triggering business-logic rules, fluctuating between 0.5-0.9% over 60 minutes
Not all fingerprint matches resulted in blocks — only those also matching business logic patterns.

The takeaway for anyone running large-scale infrastructure: defense mechanisms need the same care as the systems they protect. They require observability, documentation, and active maintenance. When protections added under pressure are left to linger, they become a different kind of threat — one that harms legitimate users instead of malicious ones. GitHub acknowledged it should have caught and removed these rules sooner, and attributed the entire cleanup to user reports from social media and elsewhere.