Inside GitHub: How we hardened our SAML implementation

A decade of SAML, and why we rebuilt trust in it

GitHub has offered SAML-based single sign-on since the 2.0.0 release of GitHub Enterprise Server in November 2014. SAML 2.0 lets enterprises connect their identity providers to GitHub products, extend conditional access policies, and manage organizations at scale. Supporting that specification means generating metadata, issuing authentication requests for service provider–initiated SSO, and — most critically — processing and validating SAML responses to authenticate users.

Those code paths sit at an unusual intersection of risk factors:

• Flaws in authentication establishment or validation can lead to authentication bypass or user impersonation.
• The code depends on XML parsing, cryptography, and dense standards like XML Signature, XML Encryption, and XML Schema.
• The attack surface is wide, since data flows through users' (and attackers') browsers before it ever reaches our servers.

That combination made our SAML implementation one of the higher-risk areas we maintain. When we launched in 2014, few libraries were mature enough for our needs. After initial experiments with ruby-saml, we built our own implementation. Over the years we hardened it through internal research and the Security Bug Bounty, but each fixed vulnerability left a lingering concern: the root causes were too varied, and the underlying complexity too deep, for point-in-time fixes to give lasting confidence.

Last year, we set out to change that. We took a hard look at our homegrown code and identified four steps to put SAML on a more sustainable footing:

1. Evaluate ruby-saml and audit it rigorously.
2. Use A/B testing to validate the new library against real traffic.
3. Tighten schema validation to shrink the input attack surface.
4. Limit vulnerability impact by using multiple parsers.

Why we went back to ruby-saml

Maintaining a security-critical protocol implementation in-house means you are also maintaining the surrounding ecosystem alone. We decided a library with strong community support — one we could contribute to alongside other developers — was the better path. We reviewed several Ruby SAML libraries and landed on ruby-saml, maintained by Sixto Martín, for three concrete reasons:

• It is widely adopted, including through omniauth-saml, across numerous critical SaaS products.
• Recent vulnerabilities were being reported and fixed promptly, demonstrating active maintenance and a working security response.
• Fixes were distributed via the GitHub Advisory Database and CVEs, with updates pushed through Dependabot — fitting naturally into our existing vulnerability management workflows.

Switching was still not a casual decision. We had years of familiarity with our own code and had sunk significant effort into finding its flaws. We did not want to inherit a new set of equally stubborn problems. So before making any change, we ran a gauntlet of validation with our bug bounty team, product security team, and the GitHub Security Lab. That collaboration — including in-depth code analysis by VIP bounty researchers like @ahacker1 and Security Lab researchers like @p- — surfaced critical vulnerabilities in ruby-saml and pointed the way toward hardening that could remove entire classes of bugs.

Still, no amount of time-bound audit is sufficient for code this complex. The next phase was about engineering process: validate the candidate library under real load, find edge cases we had missed, and reduce the size of the target we were defending.

A/B testing against live SAML traffic

GitHub.com processes roughly one million SAML payloads per business day, making it the most-used external authentication path we support. That scale demands caution: any change to this front door of enterprise authentication needs rigorous testing before it can be trusted.

To experiment safely, we used Scientist, a library we open sourced that runs a control (our current implementation) against a candidate (ruby-saml) and records observations about the differences. Scientist always honors the control's result and isolates candidate failures, which makes it an ideal tool for validating risky changes in production traffic.

Applying Scientist to SAML response validation

GitHub supports SAML against organizations and enterprises, each handled by a separate controller that implements metadata, authentication initiation, and response validation. Our focus was the Assertion Consumer Service (ACS) URL — the endpoint that processes SAML responses from identity providers. It is where the real work happens and where vulnerabilities have historically been found.

To validate ruby-saml against existing traffic, we wrapped the ACS controller logic in Scientist experiments and focused on three capabilities:

1. Granular rollout gating: In addition to Scientist's percent-based traffic control, we added feature flags so we could route our own test accounts through the candidate before any customer traffic.
2. Observability: We sent experiment metrics to Datadog and added supplemental logging for more granular validation data to help debug discrepancies between the two implementations.
3. Idempotency: Some state is tracked during a SAML flow (for example, CSRF tokens). We ensured our experiment code paths never modified that state, to avoid accidental overwrites.

The resulting experiment setup looked like this:

# gate the experiment by business, allowing us to run test account traffic through first
if business.feature_enabled?(:run_consume_experiment)
  # auth_result is the result of `e.use` below
  auth_result = science "consume_experiment" do |e|

    # ensure that we isolate the raw response ahead of time, and scope the experiment to
    # just the validation portion of response processing
    e.use { consume_control_validation(raw_saml_response) }
    e.try { consume_candidate_validation(raw_saml_response) }

    # compare results and perform logging
    e.compare { |control, candidate| compare_and_log_results(control, candidate) }
  end
end

# deal with auth_result below...

Running these experiments surfaced configuration differences between the implementations and guided how we integrated with ruby-saml to maintain behavioral consistency.

One concrete example: in September 2024, logs showed that about 3% of mismatches stemmed from SAML issuer validation discrepancies. We found that ruby-saml validates the issuer against an empty string, while GitHub has never required an issuer for all SAML configurations — when the value is blank or unset, we skip that validation entirely. To preserve this legacy invariant, we shipped a change that prevents configuring ruby-saml with blank or null issuer values, allowing the library to skip the check.

The impact of that alignment is visible below:

Once the configuration issues were resolved, we ran all production traffic through ruby-saml over an extended period. That sustained exposure let us catalog differences and investigate each for security relevance. In many cases, the new library was stricter than our own implementation — for instance, it rejected responses with multiple SAML assertions while ours accepted them. That leniency was a sign our implementation was trying to do too much. The data from these experiments let us safely augment the candidate and identify the next fronts for hardening.

Stricter schema validation

Much of the complexity in SAML processing comes from the XML itself. By tightening the schema against which incoming SAML responses are validated, we can reject a large class of malformed or attacker-crafted inputs before they ever reach deeper parsing and cryptographic logic. The narrower the input we accept, the less room there is for parser differentials and edge-case logic bugs to cause trouble.

Multiple parsers to limit impact

No matter how well a library is written or how thoroughly it is tested, vulnerabilities will still be found. To reduce the impact of those inevitable flaws, we run more than one parser over the SAML XML. If an attacker must satisfy what amounts to two independent constraints — and a vulnerability only works against one parser — the window for exploitation narrows considerably. This defense-in-depth approach means a single parser bug is far less likely to become a full authentication bypass.

None of these changes on their own would have been enough. The point-in-time audits gave us a baseline, the Scientist experiments gave us behavioral confidence under real traffic, the tightened schema shrank the attack surface, and the multi-parser design limits the blast radius of any future flaw. That combination is what let us move from a homegrown implementation we no longer trusted to a maintained library we configured and defended by design.

Why SAML validation keeps breaking

The hard part of SAML isn't the cryptography—it's the parsing. Two properties of the format conspire to make validation error-prone, and both were central to our hardening effort.

Enveloped signatures couple structure to integrity

SAML relies on enveloped XML signatures, where the <Signature> element lives inside the very data it signs. A typical verification flow looks like this:

1. Locate the <Signature> element within the <Response>.
2. Extract <SignatureValue> and <SignedInfo>.
3. Pull the <Reference> (which points to the signed element by ID) and the <DigestValue>.
4. Apply the signature's transformation rules to the <Response> and compare against the digest.
5. If the digest matches, hash <SignedInfo> and verify it against <SignatureValue> using the configured public key.

Notice the problem: to verify the signature that legitimizes the document, you have to parse the document first. The integrity of the data is tied to its structure, but that same structure determines how validation runs. This circular dependency is the root of a long line of SAML vulnerabilities, most notably XML signature wrapping attacks, which trick a parser into trusting a forged element over the signed one.

Libraries typically respond by rejecting unexpected shapes, but that still means trusting untrusted input long enough to query it—and any gap in those queries is exploitable.

The schema is more permissive than your code

SAML 2.0 responses must validate against the official XSD, but that schema is loose by design. Elements like <StatusDetail> allow arbitrary data of any type and namespace. Adding <Foo>, <Bar>, or any other element inside it is perfectly valid per the spec.

That flexibility is dangerous when signature verification depends on document structure. Consider a library that only checks the first signature it finds, assuming it belongs to the <Response>. An attacker can place a correctly signed SAML blob inside <StatusDetail> (which the schema permits) and modify the actual <Assertion> contents—since the library never verifies that signature, the tampering goes undetected. There are many documented permutations of this class of attack, and they all stem from the same lax schema.

Shrinking the attack surface with a stricter schema

We can't change the SAML spec, but we can change the schema we validate against. A stricter schema would reject ambiguous shapes before any document querying happens. The question was: what does a minimal, real-world schema look like?

We started by collecting SAML responses from test accounts of our most widely used identity providers. Entra and Okta alone drove nearly 85% of our SSO traffic. From those samples we built a bootstrapped schema, then used Scientist to A/B test it against production traffic. Iterating on failures across millions of requests, we gradually added back only the structures we actually saw in the wild.

The result is a schema that enforces the following constraints:

Signatures only where you expect them

We expect at most two signed elements: the Response and the Assertion. The official schema permits signatures in places like SubjectConfirmationData or Advice, creating ambiguous structures we never need. By removing <any> type elements, we prevent additional signatures from entering the document and cut off a whole class of signature-selection bugs.

Exactly one assertion

The SAML spec allows an unbounded number of assertions per response. We expect exactly one, and most libraries enforce this by querying and rejecting documents that contain more. Removing the minOccurs/maxOccurs attributes from the assertion choice lets us reject those documents up front, eliminating the structural ambiguity that enables signature wrapping attacks around the most critical part of the response.

Remove what you don't support

The most general advice, but the most impactful: if your implementation doesn't handle something—say, EncryptedAssertions—omit those definitions from your schema entirely. Your code can't mis-handle input that never reaches it.

Reject DTDs outright

Document type definitions are an outdated alternative to XSDs and add an unnecessary attack vector. SAML 2.0 doesn't need them, and we never observed an identity provider using one. They're disallowed in our implementation.

A stricter schema doesn't eliminate all risk—signature processing still depends on implementation—but it meaningfully reduces the parsing complexity we have to reason about.

Dual parsing: when one library isn't enough

Even after hardening validation, a fundamental truth remained: implementation bugs are inevitable. Migrating fully to ruby-saml would bring a modern, actively maintained codebase, but it also introduces unknown vulnerabilities. So we considered an alternative: instead of replacing our battle-tested library, why not run it in parallel with ruby-saml?

We implemented a dual-parsing strategy where both libraries run independently and must agree on validation before we accept a result. It's redundant by design, and that's the point:

• Defense in depth: The two libraries parse SAML differently. Exploiting both requires two independent, coordinated vulnerabilities—a significantly harder proposition.
• Built-in feedback: Disagreements trigger notifications, giving us a window into potential edge cases that could inform tighter validation logic in either library.
• Lower pressure: The original library is hardened by years of production use. Running both lets us benefit from ruby-saml's improvements without being forced to trust it completely.

There are costs: two parsers mean double the exposure to XML parsing vulnerabilities like memory corruption or XXE, and double the maintenance burden. But given that the historical critical SAML vulnerabilities are concentrated in complex validation logic, we judged the added resilience worth the investment.

A blueprint for hardening critical authentication code

What began as a straightforward migration to a new SAML library evolved into a comprehensive effort to reduce risk across our entire SAML implementation. The process offers a template for teams tackling similarly complex or security-sensitive areas of their codebase.

The key steps in our approach were:

• Upfront code review and security testing: Investing early in these practices provided confidence in the new library before it was fully integrated.
• Schema minimization with A/B testing: Using real-world data, we restricted our allowed schema to a minimal, validated version. This directly reduced the complexity of the parsing code paths, shrinking the attack surface.
• Defense in depth via dual libraries: By retaining parts of our internal implementation alongside ruby-saml, we combine the strengths of both. A single vulnerability found in either library is now less likely to be exploitable on its own.

With the system processing close to a million SAML responses daily, our logging and exception-handling strategy provides the observability needed to identify new hardening opportunities or adjust our approach reactively.

The most valuable takeaway from this project is that incremental, data-driven experiments—even those that feel like compromises at the outset—can yield unexpected and robust security outcomes. It is a reminder that broad security improvements often come from a series of measured steps rather than a single, sweeping replacement.