Why encryption enforcement needs its own system

Building encrypted transport for a globally distributed infrastructure is only half the battle; the other half is making sure nothing silently falls back to plaintext. For an environment where thousands of services handle billions of requests per second, regressions need to be caught and shut down quickly. The enforcement layer described here operates at the application level, using TLS to keep internal traffic secure—and does so transparently, without requiring application changes.

Coordinating across thousands of teams

Rolling out enforcement at this scale is as much an organizational problem as a technical one. Thousands of teams run services on the infrastructure, and the enforcement mechanism needs to be introduced without disrupting the people using those services. Clear communication about intent, timelines, and rollout strategy is essential.

Internal communication tools allowed the team to distribute information widely and consolidate feedback in one place. Communications to affected teams included:

  • A description of how enforcement impacts appear at the application layer
  • A dashboard for engineers to check whether their traffic would be affected
  • The rollout and monitoring plan
  • Dedicated points of contact and a group for questions and troubleshooting

This approach surfaced important feedback early. The team iterated on the rollout plan, dashboard requirements, and timelines based on input gathered during the process, which minimized disruption once enforcement went live.

The SSLWall design

Hardware choke points such as layer 7 firewalls are a natural candidate for transparent enforcement, but they present problems at this scale. Fine-grained rollouts are difficult to execute through deep packet inspection, and the blast radius of a configuration error at the network firewall level could take down traffic that should never have been touched.

Instead, the team built SSLWall, a host-level system that cuts off non-SSL connections across specified boundaries. The design requirements shaped the architecture:

  • Visibility into blocked traffic so service owners could assess impact and the team could proactively reach out when problems emerged.
  • Passive monitoring mode with a knob to flip to active enforcement, allowing early impact assessment.
  • Bypass mechanisms for legitimate plaintext use cases such as BGP, SSH, and approved diagnostic tools.
  • Support for protocols that do work before TLS—HTTP CONNECT and STARTTLS both perform operations in plaintext before the TLS handshake, and use cases like HTTP tunneling, MySQL security, and SMTP depend on them.
  • Extensible configuration so behavior can differ by environment and features can be rolled forward or back with minimal disruption.
  • Application transparency—no code changes or additional library dependencies, while keeping resource use and latency impact minimal.

These requirements pointed toward a host-level daemon with both user space and kernel components. The kernel side needed to inspect every connection with minimal compute overhead.

Using eBPF for connection inspection

Inspecting every connection without application involvement requires kernel context work, and eBPF provided the necessary capabilities:

  • tc-bpf: Traffic control (TC) filters using eBPF perform per-packet computation on traffic flowing in and out of the host. This approach works across a broader range of kernels in the fleet.
  • kprobes: Attaching eBPF programs to tcp_connect and tcp_v6_destroy_sock runs code in kernel context when TCP connections are established and torn down.
  • maps: eBPF provides access to arrays, bounded LRU maps, and perf events for state and data transfer.

Diagrams showing how kprobes, the tc filter, and our maps interact with one another when determining whether a connection needs to be blocked.
Diagrams showing how kprobes, the tc filter, and our maps interact with one another when determining whether a connection needs to be blocked.

The management daemon

A daemon manages the installed eBPF programs, emits perf event logs to Scribe, updates TC filters, handles configuration changes via Facebook's Configerator, and monitors health. The eBPF programs ship with the daemon, keeping releases manageable as a single software unit. Bundling also means the schema of BPF tables consulted by both user space and kernel space can be modified without compatibility concerns between versions.

Technical hurdles

Several challenges emerged during the rollout:

  • TCP Fast Open (TFO): The execution order of kprobes and TC filters exposed an issue with TFO, requiring a portion of the flow tracking code to move into a kprobe prehandler.
  • BPF program size limits: All BPF programs face size and complexity limits that vary by kernel version.
  • Performance: Optimizing the TC filter for high-QPS services with high fanout consumed significant engineering effort. Early exit conditions and using BPF arrays instead of LRUs where possible kept CPU impact trivial.

TransparentTLS for the long tail

Enforcement catches noncompliant traffic, but the team also needed a way to fix it for services that don't use common internal libraries. Torrent clients, open source message queues, and some Java applications fall into this category.

The solution, called Transparent TLS or TTLS, has three requirements:

  • Encrypt connections transparently without application changes
  • Avoid double encryption for connections that already use TLS
  • Accept suboptimal performance for this long tail

A proxy approach fit these constraints, provided application code required no modifications and configuration stayed minimal.

The technical challenge is transparent redirection of application connections to the local proxy. The cgroup/connect6 hook in BPF intercepts all connect(2) calls made by the application and redirects them as needed.

Diagram showing application and proxy logic for transparent connect.
Diagram showing application and proxy logic for transparent connect.

With the application unchanged, the BPF program makes policy decisions about proxying. For instance, TLS connections already created by the application bypass the proxy entirely, avoiding double encryption overhead.

Where the work stands

With SSLWall enforcement and TTLS in place, internal traffic is encrypted at scale. The work continues, though: new BPF facilities will become usable as old kernel support is dropped, and the transparent proxy approach can improve further by leveraging custom protocols to multiplex connections and reduce overhead.