Layering a Proxy: From Raw Packets to HTTP

Oxy, our Rust-based proxy framework, was recently introduced as the foundation for many Cloudflare services. A natural question is: how does one framework handle traffic that spans everything from raw IP packets to application-layer protocols like HTTP and SSH? The answer lies in Oxy's ability to operate across multiple layers of the OSI model, letting applications built on it choose the abstraction level they need.

Entering at Layer 3

Applications using Oxy define their ingress points, or "on-ramps," through a configuration file. One option is to accept raw IP packets at the network layer (Layer 3). This capability is essential for powering Cloudflare One, which extends private networks—offices, data centers, cloud environments, and roaming users—over the Cloudflare global network. Since these private networks operate on Zero Trust principles where every access is authenticated and authorized, supporting arbitrary IP-based protocols is a requirement.

Tenant Isolation via IP Tunnels

Simply receiving raw IP packets isn't enough in a multi-tenant environment. Every packet Oxy handles must carry context, at minimum identifying its owning tenant. Oxy introduces the concept of an IP tunnel to manage this. Context is attached to the IP packets, and applications can define, parse, and consume this context in their Oxy hooks to customize behavior.

The need for this context differs based on the traffic source. For instance, a Cloudflare Zero Trust WARP client connects through a WireGuard tunnel terminated in a Cloudflare data center. The server then opens an IP tunnel to a local Oxy instance. To transfer tenant identity, Oxy relies on a Unix SOCK_SEQPACKET socket—a reliable, connection-based, datagram-oriented channel. The first datagram carries the context, and all subsequent datagrams are raw IP packets with no overhead.

In contrast, Magic WAN on-ramps like GRE or IPsec tunnels deliver IP packets that are decapsulated by the Linux kernel. Here, there's no state between consecutive packets from the same customer, as the kernel routes them independently. Accommodating these differences, Oxy defines two tunnel types:

  • Connected IP tunnels: Context is passed once in the first datagram over a SOCK_SEQPACKET connection, as with WARP.
  • Unconnected IP tunnels: Each IP packet is encapsulated with Generic UDP Encapsulation (GUE) to carry context, using unconnected UDP sockets, as with Magic WAN.

While per-packet encapsulation consumes more CPU, it avoids IP fragmentation by precluding MTU limitations within our data centers. This is a worthwhile trade-off, as reassembling fragmented packets is a more significant tax on CPU and memory.

Flow Tracking and Routing

Once IP packets arrive, Oxy must decide their fate. The framework employs the concept of IP flows, which are inherent to most protocols: a point-to-point interaction is generally bounded in time and follows a state machine, whether known by the transport or application protocol.

Oxy performs flow tracking by parsing each packet's IP and transport headers—using the etherparse Rust crate—to derive a flow signature (source/destination IPs, ports, protocol). If the signature matches an existing flow, the packet is proxied along the already-determined path. If it's new, the upstream route is computed and memoized for subsequent packets. This is, in essence, router logic.

Tracking flows allows Oxy to expose their lifetime events to the application via hooks. Applications can use these events to apply Zero Trust policies, emit audit logs, collect metadata for billing, or compute routing decisions.

Upgrading Flows to TCP Streams

Most applications prefer operating at the application layer (Layer 7), closer to what the end-user experiences. Oxy enables this by upgrading an IP flow to a transport-layer (Layer 4) TCP stream. The challenge is implementing a performant TCP stack in userspace. Rather than using a suboptimal Rust-native implementation like smoltcp, we leverage the Linux kernel's battle-tested TCP implementation.

This is achieved by setting up a TUN interface—a virtual network device operated by user-space software—and adding an IP route to forward traffic to it. Oxy writes the raw IP packets to the TUN interface. However, the packets are dropped because customer IP addresses are nonsensical in our network. The missing step is Network Address Translation (NAT).

NAT in a Private Namespace

Oxy maintains its own stateful NAT. Each IP flow to be upgraded claims a NAT slot, and its packet addresses are rewritten to match the TUN interface's route. Once the kernel processes the NAT-ed packets, Oxy can bind a TCP listener to accept the resulting connections.

To choose NAT IPs without risking conflicts with other processes, Oxy uses Linux network namespaces. It dynamically starts an isolated network namespace for the TUN interface, where all local IP space is freely available. This involves a clever process design:

  1. The Oxy process runs in the root namespace without elevated permissions.
  2. It calls clone to create a new, unnamed user and network namespace.
  3. The parent and child processes communicate via a paired pipe.
  4. The child process brings up the TUN interface, establishes routes, and binds a TCP listener.
  5. The child passes the listener's file descriptor to the parent via SCM_RIGHTS.

This allows the Oxy process to operate in the standard namespace while accepting upgraded connections that live in a private, dynamically-created namespace.

Handling UDP

All TCP capabilities are mirrored for UDP, which is simpler because converting an IP packet to a UDP payload only requires stripping the IP and UDP headers. This is done entirely in user space within Oxy logic, bypassing the need for a TUN interface. Everything else functions the same across TCP and UDP, allowing UDP traffic to potentially be HTTPS in the case of QUIC-based HTTP/3.

Going Back Down: Downgrading Traffic

When an upstream responds, Oxy receives TCP/UDP data that must be reverted to raw IP packets for the client. This is a direct reversal:

  • For UDP, it simply adds the IP and UDP headers back to each payload.
  • For TCP, the kernel generates IP packets from writes to the upgraded socket, which Oxy reads from the TUN interface and un-NATs before sending the client.

More interestingly, applications can instruct Oxy to downgrade L4 traffic back to L3 IP flows. A practical example: a WARP client establishes an SSH session to a remote WARP device with SSH command logging enabled. The traffic flow proceeds as follows:

  1. IP packets from the WARP client on-ramp into Oxy.
  2. Oxy tracks and identifies the flow as TCP port 22, upgrading it to a TCP connection.
  3. The application (our Secure Web Gateway) parses the traffic for SSH command logging.
  4. Since the upstream is another WARP device, Oxy must downgrade the TCP connection back to IP packets for off-ramping.

Downgrading a TCP connection is more complex than upgrading. It requires a TCP client connection from the network namespace for each downgraded connection. The process uses the paired pipe to request these connections on-demand:

  1. Oxy reserves a NAT mapping for the flow.
  2. It requests the child process establish a TCP connection to the NAT-ed addresses.
  3. The kernel's TCP implementation issues a handshake, generating a SYN packet on the TUN interface.
  4. Oxy reads and un-NATs that packet before off-ramping it.
  5. The child process sends back the connection's file descriptor via SCM_RIGHTS.

The Oxy application then proxies the downgraded client connection into this newly-acquired TCP connection, yielding the raw IP packets read from the TUN interface.

Routing Packets Out and Testing

Just as traffic can enter at various layers, Oxy supports off-ramping at the same layers—as IP packets, TCP/UDP sockets, or HTTP(S) directly. The application logic determines the appropriate layer for the exit, including considerations like which public IPs to use for Internet egress.

Testing all of this requires generating raw IP packets, which is impractical for standard Rust integration tests. Instead, our tests cleverly reuse the internal library to create dynamic network namespaces and downgrade/upgrade TCP connections as needed. This allows tests to communicate via normal TCP against a downgrader running alongside the test suite, which then outputs the raw IP packets to feed into the Oxy instance being tested.

Lessons Learned

Oxy didn't start as a framework spanning the entire OSI model. It began as an HTTP proxy and gradually worked its way down the layers. In hindsight, following this path was correct: the ability to upgrade and downgrade traffic as needed is powerful, and the majority of proxying logic—socket primitives, observability, security, configuration—is shared across all layers.

Today, this design is battle-tested, powering Cloudflare One Zero Trust and plain WARP across millions of daily users exchanging most of their traffic through our global network. Our development journey continues, and further posts will explore specific aspects of the framework in more depth.