Real-Time Logs Without Leaving the Edge

When a service is down or a breach is underway, waiting for logs to land in a central store isn't an option. You need to watch events as they happen. The UNIX habit of tailing a file and grepping for a pattern doesn't translate to a distributed network spanning thousands of servers, though—you either log into all of them or ship everything to one place first.

Instant Logs is Cloudflare's answer to that gap. It streams HTTP logs straight to the dashboard in under three seconds, with filters that let you isolate a single event in a stream of millions. The engineering problem was building a pipeline that could do that without the latency of a round trip to a core data center.

Why Not Just Extend Logpush?

Cloudflare's existing Logpush product already handles log delivery to third-party analytics providers. It averages around 15 seconds of latency, but it's built for completeness and reliability—ideal for post-incident forensics, not for live debugging. The team initially tried to extend it for real-time use but quickly found the goals were incompatible. Guaranteeing complete delivery of all data to a single location inherently introduces latency. The system needed a different set of objectives:

  • Latency under three seconds from edge event to client display.
  • A data plane operating entirely at the edge, avoiding unnecessary core data center round trips.
  • Graceful handling of traffic from a few requests per day to hundreds of thousands per second.
  • User-definable filters applied before any sampling occurs.

The existing Logpush pipeline leans heavily on Kafka for sharding, buffering, and aggregation at a central location. Kafka clusters only run in core data centers though, and routing traffic there would add unacceptable delay. To keep the data plane at the edge, the team turned to Workers and Durable Objects. Workers provide an elastic, edge-native compute platform for receiving and transforming events, while Durable Objects—being globally unique—allow coordination of messages streaming from thousands of servers to a single object for aggregation and buffering before pushing to a client over WebSocket.

The Basic Architecture

At its simplest, the model involves a Durable Object acting as a session coordinator. A client opens a WebSocket connection to the object, and a web server sends log messages to it over HTTP for forwarding to the client. Because Durable Objects live at or near the data center where they're first requested, this adds almost no latency beyond a direct server-to-client payload.

Adding more servers is straightforward—they all route to the same Durable Object, which merges their streams into a single WebSocket connection to the client. But Durable Objects are single-threaded. As event sources multiply, the object eventually saturates CPU and starts rejecting requests, and even without that bottleneck, a client can't render log lines fast enough at high volumes. The solution comes in three stages: filtering, sharding, and sampling.

Filtering at the Edge

The most direct way to cut volume is to drop irrelevant events before they reach the client. Doing that filtering inside the Durable Object would only worsen its CPU saturation. Instead, the filtering pushes out to an invoking Worker. Workers scale elastically and run many filter operations in parallel as they process incoming requests to the Durable Object. The architecture begins to resemble MapReduce—map in the Workers, reduce in the object.

Sharding Across Durable Objects

Filtering isn't enough on its own. Coordinating thousands of servers sending events every second still overwhelms a single Durable Object. The solution is a sharding layer of Durable Objects—Durable Shards—that reduces the request load on the primary object. Each message is assigned a shard key that becomes part of the downstream Durable Object's name. With well-balanced keys, this reduces the load on the primary object by roughly 1/N.

Controlled Sampling With Reservoirs

The last requirement—sensible results at extreme scales—needed more than just handling throughput. Even with infinite headroom, a client cannot process millions of log lines per second. The risk was uncontrolled data shedding: if a saturated Durable Object drops connections, you lose records and can't reconstruct what the original traffic looked like.

Instant Logs instead uses reservoir sampling, which selects a fixed k items from a stream of unknown length n in a single pass. Data is buffered in the reservoir and flushed on a sub-second interval, yielding random samples at a maximum data rate of choice. This is implemented at both Durable Object layers.

The technique preserves statistical fidelity by attaching a sample interval to each line, representing the number of dropped samples for each one that passes (1/probability). Summing the intervals within a time window gives the approximate actual request count. The buffering adds slight latency, but it means nearly any event source volume can be pointed at the pipeline and handled in a controlled way.

The result is a pipeline that behaves sensibly whether it's handling a handful of requests a day or hundreds of thousands per second. Workers and Durable Objects handle the workload with no tuning required. The Instant Logs beta launches in a few weeks; Cloudflare is also hiring data engineers in Lisbon, London, Austin, and San Francisco.