Analytics Engine opens its beta with better queries and clearer errors
Cloudflare has moved Workers Analytics Engine into open beta. The service, built on Cloudflare Workers, is designed to store and query high-cardinality, high-volume time series data. Cloudflare itself relies on it to observe how customers use its products.
One internal use case is monitoring the backend behind Instant Logs, Cloudflare's live HTTP log streaming feature. That backend, itself built on Workers, receives HTTP log data from every Cloudflare server handling customer traffic and forwards it over WebSockets to a dashboard. To verify delivery is healthy, Cloudflare tracks request rates across all active Instant Log sessions and across every data center participating in the Workers network. That combination produces enormous cardinality across the data set.
Traditional metrics stores such as Prometheus are poorly suited to that kind of label explosion, so Cloudflare routes all Instant Logs backend request data into Analytics Engine instead. Operators can then query request rates by top session or data center over the previous month using the service's SQL API and render them in Grafana, with response times fast enough for interactive debugging.
Closed-beta fixes
Feedback from the closed beta praised the SQL interface, easy Workers integration, Grafana support, and the free pricing tier. The main complaint was that query failures were unreadable. The original API answered bad queries with a generic "Sorry, we were unable to evaluate your query." The beta rewrite replaces that with specific diagnostics:
> SELECT column_that_does_not_exist FROM your_dataset FORMAT JSON
now returns:
cannot select unknown column: "column_that_does_not_exist"
Cloudflare also added a comprehensive SQL reference for the API plus how-to guides, including instructions for wiring the service into Grafana.
Adaptive resolution and the _sample_interval field
To keep queries responsive, Analytics Engine draws on Cloudflare's adaptive bit rate (ABR) technology. Every query is answered using a resolution of the stored data appropriate to the time range being examined. A query spanning a month may use coarser data than one looking at the last hour, but both return correct answers within an interactive latency window.
Because a response can blend data from multiple resolutions, each event carries a _sample_interval column indicating the sampling rate of its source. An event from a data set that was sampled at 1 percent, for instance, has a _sample_interval value of 100. Reconstructing the count in the original data is a simple sum:
SELECT sum(_sample_interval) AS count FROM dataset
The open beta exposes _sample_interval directly. Future releases will add convenience functions that account for resolution differences automatically and provide confidence levels for the estimates those functions return.
What's next
Several additions are in the pipeline. Alerting on Analytics Engine data is a requested feature both internally and from developers. Later releases will also let users name the indexed fields that are currently accessed positionally as blob1, blob2, double1, double2, and so on, and query by those names through the SQL API.
The longer-term goal is a dashboard experience built into the Cloudflare dashboard so that developers don't have to operate their own visualization stack on top of Analytics Engine data. In the meantime, the service is available for trial in the Cloudflare dashboard, and Cloudflare is collecting feedback through its Discord channel.
