One SQL Interface for Everything Cloudflare Knows

Cloudflare’s network processes over a billion events per second across 330+ cities. Historically, the data behind those events lived in dozens of silos: production databases, ClickHouse clusters, Kafka streams, cloud buckets, and long-tail pipelines. Answering a basic question like "How many domains that signed up today are in the Top 100 by traffic?" required tribal knowledge about which system to query, which credentials to use, and whether the underlying data was sampled or stale.

To fix this, Cloudflare built two internal tools: Town Lake, a unified data analytics platform, and Skipper, an AI agent that runs on top of it. Town Lake provides a single SQL interface to the company's data; Skipper lets anyone ask questions in plain English and receive auditable, correct answers in seconds.

Symptoms of Data Sprawl

Cloudflare's data environment exhibited familiar hyper-growth problems:

  • Disparate systems: A product engineer investigating a customer issue might need to query Postgres for metadata, ClickHouse for analytics events, BigQuery for usage rollups, R2 for raw logs, and Kafka for real-time signals — each with its own credentials, language, and retention policy.
  • Sampled data: The analytics pipeline downsamples to handle 700M+ events per second. That works for dashboards but fails for billing or security investigations where exact numbers are required.
  • External dependencies: Parts of the internal reporting stack relied on external vendors for critical data.
  • Undiscoverable data: Knowing where "Billable Workers requests by account" lived required an obscure chain of cluster names, schemas, and customer ID translations.

Town Lake's Lakehouse Architecture

Town Lake is a data lakehouse: a query engine reading from object storage with a metadata layer that makes storage behave like a database. Its core components include:

Query engine. Apache Trino enables a single SQL query to join a Postgres table, a ClickHouse table, and an Iceberg table on R2 without materializing intermediate results elsewhere. Filters are pushed down to source systems where possible.

R2 Data Catalog. This managed Apache Iceberg service holds cold and warm data. Iceberg provides schema evolution, time travel, and partition evolution, allowing per-minute usage from last week to become hourly, then daily, as it ages. This reduces storage costs while keeping data queryable.

DataHub. The metadata catalog stores every table, column, owner, lineage edge, and glossary term, enabling answers to "what's in this table?" queries.

Lifeguard. This access control service stores rules in D1, pulls user and group membership dynamically, and renders a combined JSON policy for Trino. It also feeds access info to Skipper and the Gateway to block users early.

Skimmer. A PII detection scanner samples rows from every column and uses Workers AI to classify columns in two passes: a fast per-column classifier, followed by an agentic second pass with full table context that can query Trino directly to verify flag.

Transformer. An ELT engine built on Workflows. Users define DAGs of SQL transformations with YAML frontmatter (target table, materialization mode, dependencies, schedule); Transformer compiles and runs them on Trino with state managed by Durable Objects.

Ingestion. A long-lived Kubernetes orchestrator reads pipeline configs and spawns short-lived workers to extract from Postgres or ClickHouse, transform to Parquet, and load into R2 as Iceberg tables.

Town Lake is built entirely on Cloudflare's own platform: R2 for storage, Workers for compute, Access for authentication, Workflows for orchestration.

Default-Closed Governance

Town Lake inverts the traditional "open by default" security model. Tables are inaccessible for querying until reviewed. When a new table arrives, Skimmer scans and classifies it as pending in the central allowlist. Users cannot query it until a reviewer approves both the table and its specific columns.

This works because it is automated and self-serve. Skimmer catches obvious PII and long-tail sensitive data like API tokens and opaque IDs. Reviewers approve, override, or deny — typically in seconds. The error message for an unreviewed table is not "permission denied" but "this table needs review, click here to request one." Skipper can even suggest an appropriate RBAC group.

Schema discovery is separated from data access: users can see table names, but unreviewed columns are hidden from DESCRIBE, SHOW COLUMNS, and SELECT *, so a new unreviewed column doesn't break existing dashboards. PII is opt-in per session — Trino redacts sensitive columns by default, and users with legitimate need can flip a session bit, triggering permission checks and full query logging.

Skipper: The AI Data Agent

Skipper is a conversational agent that moves from natural-language question to validated answer grounded in Cloudflare's data, code, and institutional knowledge. Its chat interface handles queries like:

Show me the top 10 customers by R2 storage cost in the last 30 days, and the change versus the previous 30 days.

Skipper finds the right tables via DataHub search, pulls schemas and lineage, writes SQL, submits it to Trino, and displays results as a table or chart. Follow-ups carry context:

Now break it down by region, and ignore internal Cloudflare accounts.

Skipper can package charts into shareable, embeddable dashboards, build transformation graphs via Transformer, and check permissions via Lifeguard. It runs as the calling user — if you lack table access, Skipper cannot query it for you, and group membership is checked at view time, not save time.

The agent is accessible via a Worker backed by Workers AI, and its tools are exposed through an MCP server for local agentic workflows.

Grounded Context Layers

Early experiments proved that raw schema information alone causes hallucinations — invented joins, misused columns, and confident wrong numbers. The fix is layered context:

Layer 1: Schema and usage metadata. DataHub tracks column types, primary and foreign keys, and common join patterns from historical queries. Skipper's search_datasets and get_entity_details tools surface this directly.

Layer 2: Human annotations. Table descriptions, ownership, and tags like curated live in DataHub and inform Skipper which tables to prefer over scratch space.

Layer 3: Code-derived knowledge. Transformer pipelines emit per-node .meta.json documentation on every successful run. This reveals nuances like an alloc_amount column computed as billed_amount / 12 for annual; billed_amount for monthly — context that never lives in column descriptions.

Layer 4: Curated data models. Short human-written guides describe how to reason about billing, customers, and zones, with directives like "Prefer tables tagged 'curated'" surfaced as MCP resources.

Layer 5: Runtime introspection. As a safety net, Skipper can issue live queries like DESCRIBE table, SELECT DISTINCT col LIMIT 20, or SELECT COUNT(*) — used sparingly since runtime context is expensive.

Code Mode and MCP

Instead of exposing 30 individual tools over MCP, Skipper exposes just two: search and execute. The model writes a JavaScript snippet that calls the entire toolset programmatically, running in a sandboxed Dynamic Worker isolate via WorkerLoader. This lets the model express complex multi-step workflows in a single round-trip — faster, cheaper, and auditable as code.

What It Powers

Billing. The customer-facing Billable Usage Dashboard runs on a metering pipeline built on Iceberg tables in R2, queried via Trino. The dashboard API pulls the same compact (date, account_id, metric_name, usage) rows as the invoicing system, ensuring the number on screen matches the bill. Billing queries account for 53% of Town Lake's traffic — 91,760 queries from 324 distinct employees in a recent period. Legacy 200–300 line revenue rollup queries are now five lines.

Business intelligence. Questions like "top 100 customers by revenue" or "How many domains signed up today are in the top 100" resolve in about three seconds — no Jira tickets needed.

Security analytics. Bot Management queries ML scoring events with score > 0.9 in the last 48 hours filtered by ASN and geography. Threat researchers and Trust & Safety built their own query toolkits on top.

Customer support. "Find the top 100 billing support tickets from customers who have spent >$100" went from a multi-day project to a single query.

Lessons Learned

Less prompting is more. Elaborate prescriptive system prompts telling Skipper which tool to call first reduced quality. Replacing these with high-level guidance let the model reason about analytical workflows more effectively.

Tool overlap is poison. Multiple variants of similar tools confused the model. Consolidating into single-purpose tools — like fetch_results with a mode parameter for (inject / display / both) — solved it.

Code, not metadata, captures meaning. The biggest accuracy wins came from ingesting the actual SQL behind each table. Code reveals defaults, edge case handling, and fallback logic invisible in column descriptions.

Memory matters. A memory layer for corrections like "you have to filter for X like this" lets the agent improve monotonically on recurring queries instead of rediscovering lessons each conversation.

The boring infrastructure is the hard part. Trino + Iceberg is not new. The difficult work is per-row access control, default-closed allowlisting, query auditing, time-bound credentials, PII detection, idempotent ingestion, and schema evolution — the components that make the platform safe to open up.

What's Next

Cloudflare is expanding Skipper's agent surface, integrating it deeper into internal chat and ticketing systems. The Transformer pipeline is being invested in heavily so any team can define a curated dataset with SQL files and a .meta.json, deploy it as a Workflow, and have it automatically surface in DataHub and Skipper. As R2 SQL matures, more of Town Lake's workflow is planned to move onto it.