A Low-Cost Protocol Testbed for Real-World Transport Development

Transport protocol development demands rigorous testing across varied network conditions. At Cloudflare, work on QUIC and HTTP/3 has driven the creation of a dedicated lab environment built from affordable hardware and open-source tools, enabling reproducible experiments with fine-grained control over network parameters.

QUIC is a secure, multiplexed transport protocol designed to outperform TCP under certain network conditions. The protocol family spans multiple layers: the transport layer defines packet formats and the basic state machine; recovery and congestion control govern reliability; security is built on TLS 1.3; and the application mapping is HTTP/3.

The transport and recovery layer is the foundation—it determines the wire format, connection establishment, secure handshakes, reliable data delivery, and reactions to packet loss or reordering. It also enforces flow and congestion control to coexist with other protocols on the network. Confidence in this layer is prerequisite to building higher-level applications such as HTTP/3.

Development typically begins on localhost, then moves to virtual machines or containers for early interoperability testing. However, these environments are often noisy or restricted to loopback networking. A dedicated physical lab provides a quiet, isolated network exclusively for protocol analysis, with the flexibility to test a wide range of scenarios—from legacy EDGE and current LTE networks to emerging 5G and WiFi conditions.

The QUIC testing lab at Cloudflare was built with four primary objectives:

  • Iterative development support: validate packet construction against spec, test under moderate load and extreme conditions like low bandwidth and high loss, and run reproducible experiments to uncover unexpected issues.
  • Deep debugging capabilities: record extensive client and server logs beyond just packet captures to understand the rationale behind every transmission—whether triggered by application data, retransmission of lost packets, or a loss probe sent to test network health.
  • Performance comparison: benchmark new protocols against TCP, prior versions, and variants with different congestion control algorithms, timeouts, or buffer sizes.
  • Rapid fault detection: verify each test's integrity via checksums comparing original and downloaded files, or by monitoring protocol and API error codes.

A dedicated lab on separate hardware offers several operational benefits: it can be configured without public Internet access for safety and quiet, consoles are readily available for maintenance, and alternative CPU architectures can be exercised. The lab at Cloudflare uses Raspberry Pi devices as clients and origin servers—their ARM architecture parallels modern smartphones. Real smartphones running Android or iOS can also be integrated, supporting both WiFi and wired Ethernet testing for consistency.

Lab Architecture

The physical lab interconnects several components via Ethernet switches:

  • Origin: serves HTTP and HTTPS test objects via a standard web server.
  • Client: downloads objects from Origin or Edge using curl for TCP/HTTP and quiche's http3_client (with modifications) for QUIC.
  • Edge: runs Cloudflare's web server serving HTTP/HTTPS over TCP and QUIC; it uses the same Linux kernel as production machines.
  • Traffic Shaper: sits between Client and Edge/Origin, controlling bandwidth, latency, and packet loss; currently implemented on FreeBSD with ipfw and dummynet, though Linux's netem is an alternative with additional simulation features.

The lab can run plaintext HTTP tests, but the primary focus is HTTPS over TCP versus HTTP/3 over QUIC. Since QUIC runs on UDP, the traffic shaper must govern both TCP and UDP flows.

Automation and Metrics

A test script on the Client orchestrates batch runs with parameterized configurations:

  • Network conditions: bandwidth, latency, packet loss (upstream and downstream), and queue sizes—e.g., simulating an LTE profile with 50ms RTT and 22Mbps bandwidth.
  • Object sizes: from 1KB to 32MB.
  • Protocols: HTTPS (TCP) and QUIC (UDP).
  • Test repetitions: number of runs and requests per connection.

Results are output as CSV files for import into Google Sheets, Excel, or jupyter notebooks, and can also be posted to a ClickHouse database for querying and visualization. Full test suites spanning multiple network profiles and object sizes can take hours, so a limited smoke test (e.g., LTE only) is used for rapid debugging cycles.

Comparison charts visualize total transfer time in milliseconds for TCP versus QUIC across different simulated network conditions, offering direct insight into protocol performance during development.

Testing on Real Mobile Devices

Mobile devices are central to daily network usage, making protocol testing on them essential for app performance. Adding a real smartphone to the testbed provides invaluable insight into the distinct network stacks of iOS and Android, informing new protocol designs.

In a parallel lab setup, a smartphone acts as the client connected both wired and wirelessly. A Linux netem-based shaper between client and server emulates real-world networking profiles, while a standard web server serves static files. The test app, built with proprietary proxy software, handles data over the new transport protocol and also issues HTTP requests over TCP for comparison.

Tests issue multiple HTTPS requests of varying object sizes, both sequentially and concurrently, over TCP and QUIC. Total transfer time is measured for each request. Results from an LTE profile show TCP over the native OS stack versus QUIC over the Cloudflare stack at different concurrency levels.

Packet capture on mobile devices, however, is nontrivial. On iOS, Apple's rvictl enables external interface capture but suffers from inaccurate timestamps—problematic when diagnosing millisecond-level events. Jailbroken iPhones and rooted Android devices allow loobpack interface captures via tcpdump, but keeping a device jailbroken requires disabling all automatic updates. Packet captures from such devices have proven crucial in identifying bottlenecks and understanding platform network behavior—for example, revealing a mysterious 5–7ms delay for packets traversing the loopback interface on iOS.

Why a Custom Testbed Wins

Building protocol implementations in the lab is one thing; seeing how they behave under real-world chaos is another. Off-the-shelf network simulators often abstract away the very details that matter—buffer bloat, packet reordering, and sudden bandwidth shifts. A fixed, synthetic environment also makes it hard to compare results across teams or to replay a specific failure mode that only appears under particular conditions.

An in-house testbed offers three concrete advantages:

  • Reproducibility: The same physical setup can be configured to mimic different network profiles, from low-latency data centers to high-loss mobile links.
  • Extensibility: New protocol versions or congestion control algorithms can be dropped into the existing infrastructure without rewriting the whole harness.
  • Cost efficiency: Using commodity hardware and open-source tools keeps the barrier to entry low while still providing enough fidelity to catch subtle bugs.

Anatomy of the Testbed

The core components are straightforward: a pair of end hosts running the implementation under test, separated by a network emulation layer. This layer is where the magic happens—it applies controlled delay, loss, and bandwidth constraints to every packet that crosses it. The emulator runs on its own dedicated machine to avoid CPU contention affecting the results.

Traffic generation is handled by a separate tool that can produce both bulk transfers and interactive streams, matching the mix of workloads that QUIC and HTTP/3 are expected to carry. Measurement and logging are centralized, so every run produces a consistent set of metrics: throughput, latency percentiles, and loss recovery rates. The whole setup is orchestrated by a simple script that resets the environment between runs, ensuring that tests are independent and repeatable.

Validating the Setup

Before trusting any results, the team verified that the testbed behaves as expected. A baseline test using TCP produced throughput curves that matched theoretical limits for the configured bandwidth and delay, confirming that the emulation layer wasn't introducing artifacts. Similarly, loss profiles were checked to ensure the emulator dropped packets at the configured rate and not in some unintended pattern.

This validation step is critical. Without it, a developer might chase a performance regression that is actually a side effect of the test harness rather than a change in the protocol stack. The authors of the testbed recommend running these sanity checks whenever the network topology or emulator configuration changes.

Lessons for Future Protocol Work

The effort spent building this testbed pays off not just for QUIC but for any future transport protocol work. The patterns established here—modular emulation, automated orchestration, and rigorous baseline checks—are transferable. When the next protocol iteration needs testing, the infrastructure is already in place and proven.

This approach aligns with the broader mission of pushing internet standards forward not just in theory but in practice, by giving developers the tools to observe how their code behaves under the messy conditions of the real world.

Acknowledgments to SangJo Lee, Hiren Panchasara, Lucas Pardue, and Sreeni Tellakula for their contributions.