Oxy: Fish/Bumblebee/Splicer subsystems to improve reliability

From Monolith to Microservices: How Cloudflare Split Its Proxy

Building proxy applications on top of Oxy means preparing for enormous traffic volumes while remaining resilient to crashes and reloads. As the framework grows more complex, these requirements pull in different directions. While migrating WARP to support soft-unicast (Cloudflare servers don't own IPs anymore), Cloudflare added features that ballooned both code size and the state needing preservation across process upgrades.

The solution was to break the large proxy process into specialized services, each following the Unix philosophy of doing one thing well. Three services handle distinct responsibilities: Splicer pipes data between sockets, Bumblebee upgrades an IP flow to a TCP socket, and Fish handles layer 3 egress using soft-unicast IPs. Together, they improve system reliability and efficiency as WARP transitions to soft-unicast.

A Service for Every Layer

Splicer: Socket Relaying as a Service

Most transmission tunnels in Cloudflare's proxy forward packets without modification: read from one socket, write to another. This pattern appeared repeatedly across projects, each with its own tweaks for buffering, flushing, and connection termination—and each needing to coordinate with process restart handling.

Splicer externalizes this task. Applications hand two sockets to Splicer, which runs the long-lived relay. The calling application no longer manages connection lifetime across restarts. When the relay finishes, Splicer returns the original sockets along with the attached metadata, allowing the caller to inspect final socket state—for instance, via TCP_INFO—and finalize audit logging.

Bumblebee: From IP Packets to TCP Sockets

Cloudflare's on-ramps are often IP-based (layer 3), but internal services typically operate on TCP or UDP sockets (layer 4). Bumblebee bridges the gap by creating a kernel TCP socket from raw IP packets—no user-space TCP stack required.

The service spawns a thread in an anonymous network namespace using the unshare syscall, applies NAT to the IP packets, and uses a tun device to perform TCP three-way handshakes against a listener. Callers simply pass a socket carrying the IP flow; Bumblebee returns a proper TCP socket. Since Bumblebee handles the IP side while Splicer handles TCP, a proxy restart doesn't break in-flight connections.

Fish: Soft-Unicast Egress Without Conntrack Hell

Fish forwards IP packets using soft-unicast IP space without upgrading to layer 4. The previous approach—iptables with conntrack—became unwieldy with many possible egress IPs and variable port assignments. Conntrack is highly configurable, but applying configuration via iptables rules requires careful coordination, and debugging iptables execution is difficult. Failure modes grew arcane when conntrack couldn't rewrite packets to an exact IP or port range.

Fish rewrites packets and configures conntrack directly via the netlink protocol. A proxy application sends a socket of IP packets with the desired soft-unicast IP and port range. Fish then forwards the packets to their destination, ensuring a unique five-tuple in the root network namespace regardless of the client's chosen source IP. Fish's internal state survives its own restarts.

The Unix Philosophy, Applied

Rather than bolting these functions onto the proxy, the services stand alone, each with its own restart life-cycle. That yields a distinct advantage: failure cases are easier to understand in a small system, making it practical to design for reliable behavior. Fixing a small service well improves the reliability of everything built on it.

While the three services have different business logic, they share a core pattern: receiving sockets (file descriptors) from other applications, enabling those applications to restart without dropping connections. The services themselves can restart the same way.

Passing File Descriptors

Interprocess communication happens over Unix Domain Sockets, which support passing file descriptors between processes—essential for both the architecture and graceful restart.

Two approaches exist: the pid_getfd syscall or SCM_RIGHTS. SCM_RIGHTS fits better here because use cases center on the proxy "giving" sockets rather than services "taking" them. The syscall approach also requires special permissions and a signaling mechanism to indicate which descriptor to take.

Internally, Cloudflare uses a library called hot-potato since production runs stable Rust. Nightly users might prefer the unix_socket_ancillary_data feature. Several production considerations matter when using SCM_RIGHTS:

• A maximum of 253 file descriptors can be passed per message (SCM_MAX_FD, set since kernel 2.6.38)
• Peer credentials are worth retrieving for observability in multi-tenant settings
• SCM_RIGHTS ancillary data forms a message boundary
• Any file descriptor type can be sent, not just sockets—combining this with memfd_create bypasses buffer size limits without length-encoded frames, enabling zero-copy message passing

Graceful Restarts via State Passing

The general strategy for graceful restart was covered in "Oxy: the journey of graceful restarts." Unlike NGINX-style reloads that leave lingering processes handling pending requests—problematic for observability and prone to performance degradation after repeated reloads—these microservices pass state to the new process.

Pending requests are paused, collected, and transferred to the new process immediately on startup. This demands more complexity than keeping the old process alive but avoids lingering processes. When a service receives an upgrade request (typically SIGHUP): pause all tasks, wait for task groups to pause, and send them to the new process.

Go solves this with WaitGroup; Rust offers channel-based implementations covered earlier, the waitgroup crate using AtomicWaker, or a JoinSet. A JoinSet keeps results for all requests, which grows memory pressure under load. Cleaning up the JoinSet as requests are processed reduces that overhead.

    let mut task_group = JoinSet::new();

    loop {
        // Receive the request from a listener
        let Some(request) = listener.recv().await else {
            println!("There is no more request");
            break;
        };
        // Spawn a task that will process request.
        // This returns immediately
        task_group.spawn(process_request(request));
    }

    // Wait for all requests to be completed before continue
    while task_group.join_next().await.is_some() {}

Cancellation Safety

Passing pending tasks promptly on upgrade signal requires graceful shutdown. Tokio's shutdown tutorial covers channel-based cancellation, but the tasks being paused must be cancellation-safe. Paused results collect into the JoinSet and are handed to the new process via file descriptor passing.

For a service like Bumblebee, the paused state includes environment file descriptors, the client socket, and the socket proxying the IP flow. The NAT table is too large for socket buffers, so it's encoded into an anonymous file descriptor—which then transfers to the new process like any other.

Why the Split Is Worth It

Splitting a complex proxy into focused microservices lets each component have its own lifetime and failure domain. The architecture incurs costs—distributed tracing and interprocess communication—but the gains in performance, maintainability, and reliability outweigh them. Future work will cover debugging techniques for large codebases with complex service interactions, using tools like strace and eBPF.