When a Root Certificate Expiry Broke Internal TLS, Not Edge Traffic
On September 30, 2021, Let’s Encrypt’s old root certificate expired at 10:00 a.m. Eastern Standard Time. Shopify had prepared for this. As a Let’s Encrypt sponsor since 2016, Shopify’s edge infrastructure had been updated to meet the new requirements, and the cross-signing of the new root by the old one ensured clients that hadn’t yet adopted the new root could still validate certificates. With Let’s Encrypt-issued certificates expiring every three months, any certificate in use by then would be signed by the new cross-signed root and valid under either trust chain. The expectation was that nothing would break.
Fifteen minutes later, monitors began reporting certificate errors. The failures were not at Shopify’s edge—the boundary between the public and Shopify’s platform—but between internal services. A member of Shopify’s traffic team, which handles certificate provisioning and the safe routing of traffic into the platform, joined the incident response.
Locking Deployments and Tracing the Error Source
The first response was to lock deployments of the Shopify monolith using spy, Shopify’s chatops tool, while responders connected to containers to inspect what was happening. Initially, the team examined the deployments that coincided with the onset of the issue. Those changes had no relation to service interconnection or certificates, so the expiry of the Let’s Encrypt root certificate began to seem like more than a coincidence—especially since the errors appeared right after the expiry time.
The errors could not be reproduced in a browser or with curl. Using openssl, however, the team could observe the old root certificate’s expiration:
The failures were client-specific, appearing across numerous services using different configurations and libraries. For several services, the errors surfaced from an internally-built library that services use to check authentication against Shopify. That library depends on rack-oauth2 and openid_connect, both of which—directly or indirectly—depend on httpclient, a Ruby library that provides functionality similar to libwww-perl (LWP).
Further investigation revealed that google-api-client was also producing errors, and the same dependency chain led back to httpclient. Looking closely at how httpclient handles certificate validation, the team found the problem:
Code snippet from httpclient/nahi
httpclient embeds a version of the root certificate store that was six years old at the time of the incident. Reference root certificate stores are generally updated every few months. Because the embedded store did not trust the new root certificate directly, nor the old one after its expiration, TLS validation failed for any new connection—even though Let’s Encrypt had done everything correctly on the CA side.
Emergency Fix and Long-Term Mitigation
The fast fix was simple. Shopify forked the httpclient repository and created a branch that overrode cacert.pem with a current root certificate bundle. Deployments were updated to use that branch, canaries confirmed the fix worked, and automation generated pull requests across all affected repositories.
Overriding cacert.pem was always meant to be a temporary solution. It was chosen because it would work automatically across all deployments without requiring additional changes. To support the fix and prevent recurrence, Shopify built systems that track root certificate changes and automatically update them in its fork when necessary. The long-term approach would be to use the system root certificate store, which requires reviewing the root stores across all runtime environments.
The 15-Minute Delay and the Real Damage
The delay between the expiration and the onset of errors puzzled the team. The reason lay in HTTP keep-alive. Connections are kept alive as long as they are actively used, and TLS validation happens only when a connection is initialized; trust is maintained for the lifetime of that connection. Given Shopify’s traffic volume, existing connections to other systems stayed alive. The issue only surfaced when Kubernetes pods were recreated for deployments, forcing new connections that failed TLS validation—hence the 15-minute gap between expiry and symptom.
The incident affected more than Ruby applications with indirect dependencies on httpclient. Data-backed services stalled when the applications providing their data were disrupted. Product recommendations disappeared temporarily, marketing campaigns were throttled, and order confirmations were delayed because risk analysis could not run.
The Shopify monolith itself was not fully affected. Only the canaries—servers that receive changes first for production testing—saw errors. The decision to lock deployments was key: it stopped pod cycling for the monolith, keeping the current version running and preventing what could have become a global outage. September 30, 2021 was a near miss, and it came down to a stale root certificate store embedded in a widely used dependency.
