Why index logs with Durable Objects
Running a traditional database to index logs is often impractical — it means standing up infrastructure, managing capacity, and paying for idle resources. The Cloudflare R2 Log Retrieval API takes a different route: it uses Durable Objects to maintain index state and R2 as the storage layer. That removes the operational burden while keeping lookup costs low.
The index itself is a forward-index that maps a batch of logs (the “document”) to the RayIDs it contains (the “words”). In this case a RayID is the unique identifier for an HTTP request. The structure lives in the transactional key-value storage of a Durable Object:
storage.put(batchName, rayIds)...for writes and storage.get(batchName) for reads. Each key is a batch file name, each value is a JSON array of RayIDs.
Streaming data into the index
Log batches stored in R2 are compressed, often at 30–100x ratios. Reading a whole batch into memory would risk out-of-memory errors in a Worker. The solution is to process the data as a stream pipeline using the Streams API, which supports both byte-oriented streams (for compression/decompression) and value-oriented streams that operate on JavaScript values like strings and objects.
When an indexing request arrives, the Worker streams the batch from R2 through a chain of TransformationStreams that decompress the data, decode bytes into strings, split records on newlines, and gather all RayIDs. Once collected, the RayIDs are written to the Durable Object's storage.
async function index(r2, bucket, key, storage) {
const obj = await getObject(r2, bucket, key);
const rawStream = obj.Body as ReadableStream;
const index: Record<string, string[]> = {};
const rayIdIndexingStream = new TransformStream({
transform(chunk: string) {
for (const match of chunk.matchAll(RAYID_FIELD_REGEX)) {
const { rayid } = match.groups!;
if (key in index) {
index[key].push(rayid);
} else {
index[key] = [rayid];
}
}
}
});
await collectStream(rawStream.pipeThrough(new DecompressionStream('gzip')).pipeThrough(textDecoderStream()).pipeThrough(readlineStream()).pipeThrough(rayIdIndexingStream));
storage.put(index);
}Narrowing the search space
If a zone produces one batch per second, that's over 86,400 batches per day — far too many keys to iterate through on every lookup. Two characteristics of the data keep the search manageable:
- A RayID is a 16-byte hex value whose first 36 bits encode a timestamp. That timestamp is a lower bound on when the request completed.
- Logpush batch names encode the time range of the logs they contain.
For a request logged on November 3 at 16:00:00 UTC, the lookup only needs to consider batches under the prefix 20221103 whose recorded time range falls between 16:00:00 and 16:05:00. Analysis of the logging pipeline showed that 95% of RayIDs appear in a batch produced within five minutes of request completion.
This narrows the candidate set from tens of thousands of batches to a small handful. The index is then checked by iterating over just those batch-name keys.
If a match is found, the corresponding batch is streamed back from R2 through another TransformationStream pipeline that filters out non-matching records. If nothing matches, the API returns an error indicating the RayID could not be found.
async function lookup(r2, bucket, prefix, start, end, rayid, storage) {
const keys = await listObjects(r2, bucket, prefix, start, end);
for (const key of keys) {
const haystack: string[] | undefined = await storage.get(key);
if (haystack && haystack.includes(rayid)) {
return key
}
}
return undefined
}Current limits and roadmap
RayID lookup is currently available for HTTP Requests and Firewall Events, with Workers Trace Events support on the way. Planned work includes indexing additional fields such as status codes and hostnames, exposing retrieval through the dashboard, and supporting more complex filters and queries against logs.
