When Deploying GitHub Requires GitHub

GitHub is its own biggest customer, and that comes with a hidden risk: the company hosts all of its own source code on github.com. If github.com goes down, GitHub engineers lose access to the code they need to fix it. A mirror of the code exists for fixing forward, and built assets are available for rollback, but that only solves the first circular dependency.

Deployment scripts themselves can create their own circular dependencies. A script might pull a release binary from GitHub (a direct dependency), a servicing tool already on disk might check GitHub for updates (a hidden dependency), or a script might call an internal service that in turn fetches from GitHub (a transient dependency).

GitHub's old approach put the onus on every team owning stateful hosts to manually review deployment scripts for these dependencies. In practice, dependencies often went unnoticed until an incident made them painfully obvious, delaying recovery. Blocking github.com entirely from the hosts wasn't an option: these machines serve production traffic even during deploys and drains, and they need that access. So GitHub turned to eBPF.

Filtering at the cGroup Level

eBPF allows custom programs to be loaded into the Linux kernel and hooked into core primitives like networking. The relevant hook here is BPF_PROG_TYPE_CGROUP_SKB, which can intercept network egress from a specific cGroup. A cGroup is a Linux primitive—heavily used by Docker but not limited to it—that provides resource isolation for sets of processes. No Docker is required to create one and move processes into it.

The idea was straightforward: create a cGroup, place only the deployment script inside it, and use eBPF to limit that script's outbound network access while leaving the rest of the system untouched.

From IP Lists to DNS-Based Blocking

The initial proof of concept was written in Go using the cilium/ebpf library, which simplifies reading, modifying, and loading eBPF programs. The CGROUP_SKB hook, however, operates on IP addresses. Given the breadth of GitHub's systems and how fast they change, maintaining an up-to-date block list of IPs would be impractical.

A second eBPF program type solved that problem. BPF_PROG_TYPE_CGROUP_SOCK_ADDR can hook socket creation syscalls and even rewrite the destination IP. GitHub used this to intercept DNS queries from the deployment cGroup and redirect them to a userspace DNS proxy running on the host. The proxy evaluates each requested domain against a block list and communicates with the CGROUP_SKB program via eBPF Maps to allow or deny the request.

Logging the Culprit

Once the basic filtering worked, GitHub wanted more: could it correlate a blocked DNS request back to the specific process that triggered it?

Inside the CGROUP_SKB program, GitHub pulled two pieces of data from the skb_buff—the DNS transaction ID and, via bpf_get_current_pid_tgid, the process ID that initiated the request. This information was placed into another eBPF Map tracking DNS Transaction ID → Process ID.

Because all DNS calls are redirected to the userspace proxy, the proxy can inspect each transaction ID, find the domain being resolved, and look up which process made the request. Reading /proc/{PID}/cmdline reveals the full command line that triggered the request, producing a log line that lets the owning team know exactly what was blocked and why.

What the Tooling Now Provides

The resulting system gives GitHub the ability to:

  • Conditionally block domains that would create circular dependencies, only for deployment scripts.
  • Inform the owning team which command triggered a blocked request.
  • Provide an audit list of all domains contacted during a deployment.
  • Use the same cGroups to enforce CPU and memory limits on deploy scripts, preventing runaway resource usage from impacting other workloads.

After a six-month rollout, the circular dependency detection is live in production. If a team adds a problematic dependency, or an existing binary tool takes on a new dependency, the tooling flags it automatically. The result is a more stable GitHub and faster mean time to recovery during incidents, since circular dependencies are caught before they can delay a fix.

GitHub acknowledges this isn't the end of the story: there are still ways for circular dependencies to slip through, and the tool will be improved as those edge cases surface. For those interested in exploring similar eBPF-based tooling, the cilium/ebpf examples and docs.ebpf.io are good starting points. Open source tools like bpftrace for deep tracing and ptcpdump for TCP dumps with container-level metadata offer a lower-friction way to start working with eBPF today.