Instrumenting the instrument: monitoring Cloudflare's SQL API
Workers Analytics Engine, which entered open beta in September, lets developers generate time series analytics with high dimensionality and cardinality from Workers code. Cloudflare's own engineering team has been using it to monitor the product itself — specifically, the SQL API that sits in front of Analytics Engine's event data.
The SQL API parses user queries, transforms and validates them, then executes them against backend database servers. Every one of those requests now produces an event written back into Analytics Engine, giving the team both classic observability data — request counts, error rates, response times — and product insight into how customers are actually writing queries. Since Analytics Engine handles high-cardinality data, the team can aggregate at scale and then drill into individual events without standing up separate metric pipelines.
What gets logged
The instrumentation written into the SQL API follows the same pattern Cloudflare exposes to its users. Each event records the fields the team wants to report on — from response times and status codes to the SQL dialect features being exercised. With the data in Analytics Engine, any query can be run against it, which is more flexible than predefined, purpose-specific metrics.
Performance observability
One of the first views the team watches is backend database cluster performance:
The p99 response time occasionally spikes to roughly 300ms, while average backend response stays under 100ms. That visualization is itself generated from an Analytics Engine SQL query:
Analytics Engine supports large datasets — trillions of data points — through a sampling method called Adaptive Bit Rate (ABR). Queries over very large data volumes may return sampled events to keep response times reasonable; with more typical data volumes, queries run against all events. For now, query authors have to account for sampling in how they write queries, though Cloudflare is exploring making that automatic.
Product insight from error data
Beyond performance, the same events feed product decisions. The SQL API supports a fairly standard SQL dialect but isn't feature-complete, and unsupported constructs return structured error messages. Those errors are captured as Analytics Engine events. Aggregating them reveals which missing features customers are hitting most:
Error messages follow the pattern type of error: more details, so grouping by the portion before the colon yields a count of each error type and how many users it affected:
With a conventional metrics system, each error type would need its own metric, and reporting that many metrics from each microservice creates scalability problems. Analytics Engine absorbs that because it's built for high-cardinality data.
The same property makes it easy to investigate spikes. When SQL errors jump, the team can pull up which customers are affected and what they were trying to do:
Findings and next steps
The usage data has already shaped the product roadmap. The error aggregates show frequent attempts to select columns — largely usability issues coming from the Grafana plugin. Adding support for the DESCRIBE function should address that, since the plugin currently lacks a way to understand table structure. Grafana plugin improvements are on the roadmap.
Queries against time ranges older than retained data point to demand for longer storage. Retention was recently extended from 31 to 92 days, with further extension under consideration. A large share of errors also appears to stem from common SQL syntax misunderstandings, suggesting that better documentation examples and error explanations would help users troubleshoot.
The same data-driven approach is open to any Cloudflare customer. Analytics Engine is in open beta with free 90-day retention.
