A decade of TLS hardening at Meta

Meta runs a globally distributed edge network that serves as the entry point for user traffic into its infrastructure. TLS termination happens at the edge, with the Proxygen L7 load balancer handling handshakes. Over time, the company has reduced how long certificates remain valid, and its latest step is short-lived certificates (SLCs) with a private key exposure window of just 10 days, rotated daily.

The move to SLCs stems from the attack surface of traditional, long-lived certificates. If a private key is compromised, the window in which it can be abused is now measured in days rather than months or years. This approach also avoids the latency costs of remote signing while preserving the benefits of keeping sign operations at the edge.

Shifting the architecture

Earlier iterations of Meta's TLS infrastructure had Proxygen load both certificates and private keys into memory to complete handshakes. That design exposed keys to vulnerabilities such as Heartbleed. The company then introduced an OffloadService, a locked-down environment separate from the external internet. This service holds private keys in memory and offloads TLS sign operations from Proxygen.

Moving signing to OffloadService decoupled the load balancer from the TLS infrastructure and allowed certificate rotation at runtime without a service push. Still, keys remained resident in memory for extended periods, which kept the risk of exposure higher than desired.

Why not revoke or remote-outsource?

Meta evaluated several alternatives before settling on SLCs. Classic revocation mechanisms, such as certificate revocation lists (CRLs) and the online certificate status protocol (OCSP), require access to a distribution point or responder for each secure channel setup. That adds latency, increases failure scenarios, and makes the overall system more complex and less reliable than one without revocation checks.

Remote offload was also tested: private keys for high-value certificates stayed in datacenter regions, with sign operations performed by RPC from the edge. This ran smoothly for six months in a few clusters, but those clusters were well connected to their nearest datacenter. The architecture proved vulnerable to datacenter outages and backbone issues, so it was not rolled out further.

Delegated credentials were another experimental path, but broader browser adoption is needed for that to have an effect across the full user base.

Building the SLC pipeline

Meta has progressively shortened certificate lifetimes from years to months, and now to days. Certificates are issued every day with a 10-day remaining validity when they are pushed to the edge. This step change required a more robust and mostly flawless automation pipeline, since a single-day failure to rotate would cause service disruption.

The main system requirements were:

  • External dependencies off the critical path: Certificates are issued by third-party CAs, but with a lifespan of a few months. They are only distributed 10 days before expiration, so even if a CA has operational issues, issued certificates are available well in advance.
  • Key lockdown: In the CA-issued bundle, private keys sit in a locked-down secret store inaccessible to edge services. Ten days before expiration, secrets move to a store accessible only to OffloadService.
  • Backup via future bundles: Because future credentials are issued in advance, they act as a fast-rotation backup in case current private keys are compromised.
  • Transparency for internal users: Certificate validity details remained hidden from non-security teams. This required a major update to the certificate management service, but allowed new sites to bootstrap an SLC series automatically without extra requestor effort.
  • Reliability at scale: With high certificate volume and time-sensitive rotation, the end-to-end flow needed to minimize manual intervention.
  • Clock skew tolerance: The 10-day remaining validity buffer is assumed to cover clock skew on 'valid_from' dates. No noticeable increase in TLS handshake errors appeared during deployment, and post-deployment client data analysis confirmed the shorter validity had no adverse effect.

Handling the load and rollout

For longer-validity certificates, rotation involved fewer than 10 certs at a time. With SLCs, every certificate renews daily, increasing the load on certificate issuance and distribution by roughly 10x. During the daily rotation, OffloadService fetches private keys from the secret store to load into memory. The simultaneous fetch requests, around 100x the previous spike, overloaded the store. This was addressed by improving caching and staggering OffloadService secret fetch requests. A prev-current-next model handles any synchronization delays during rotation.

Issued certificates are grouped by expiration date into "cert bundles" (e.g., all certs expiring on April 11 form a bundle named `expires_2023_04_11`). A ConfigBuilder picks a candidate bundle daily with a 10-day exposure window, validates and tests the certificates, and updates the OffloadService configuration that is picked up at runtime. OffloadService continues signing with both old and new private keys briefly to prevent transient sign errors during rotation.

The rollout happens in stages. ConfigBuilder first canaries the new configuration to a limited set of Proxygen instances to check handshake status. Since that smaller test cannot cover all edge cases — such as missing certs or version incompatibilities on specific machines — the bundle is then pushed to a wider deployment of about 100 machines. After baking for a day, the bundle goes out to the entire edge fleet.

What's next

Meta has cut certificate lifetime from years to days and continues to harden the pipeline. Future work includes further reducing the certificate lifetime and strengthening secret management with hardware-based techniques. The effort was led with partners including Xiangyu Bu, Puneet Mehra, Anirudh Ramachandran, and Kai Yuan Thng.