Traditional deployments assume a small number of large database clusters per application, usually paired with a connection pooler. AI-assisted development and vibe coding platforms invert that assumption, pushing applications toward many small databases: one per user, per application, per preview environment, or per tenant.

D1 is Cloudflare's managed, serverless database built on SQLite, and the one-database-per-tenant pattern suits it well. It lets you create millions of databases and pay only for what you use. The question is how to scale the control plane for millions of databases without routing every query through a centralized bottleneck.

The numbers so far: D1 has passed 5.5 million databases, with 15x growth in live non-deleted databases over the past year and 65x over two years, both as of October 8, 2026.

Daily active databases — those that executed at least one query that day — grew 18x over the past year, as of the same date.

Platforms including ChatGPT Sites build on D1 using the same tools available to any Cloudflare user: "D1 gives our users a simple, scalable way to add features like custom metrics, leaderboards, and comments to their dashboards, games, and tools," said Jon Abrams of ChatGPT Sites at OpenAI. Throughout this work, the team load-tested against a target well above expected growth to decide where optimization effort should go.

Design rules

The architecture below follows a set of principles that apply to anyone building a control plane and data plane on the Developer Platform:

  1. Design every process for failure and idempotency, so the system self-corrects and keeps making progress.
  2. Keep centralized single-region components out of any hot path for data plane operations — in D1's case, every query request.
  3. Use Workers KV, the globally distributed KV store, for critical routing metadata so cache misses only rarely reach central databases.
  4. Reach central Postgres through Hyperdrive for connection pooling and optimized routing and caching, using read replicas wherever possible.
  5. Precompute aggregations periodically in the database rather than scanning during request processing.
  6. Rely on eventual consistency and queuing between components that don't need strong consistency.
  7. Load-test with realistic workloads to find bottlenecks, and optimize those first.

Component overview

The diagram omits some secondary processes, but it covers the pieces behind D1's public feature set. Each D1 database is backed by one SQLite Durable Object, which is what makes creating millions of them practical.

The "HTTP API" box stands for every API surface: the Cloudflare dashboard, Wrangler, and the Cloudflare CLI. All API requests pass through Cloudflare's edge gateway and are served by the D1 Edge API Worker. The Worker exposes query endpoints for executing SQL, plus CRUD endpoints for creating, deleting and listing databases along with import/export and time travel.

CRUD endpoints query a Postgres cluster in central data centers ("Core") over Hyperdrive, in two configurations: one with caching enabled targeting Postgres read replicas, and one with caching disabled targeting the primary. Depending on request type, the Edge API Worker may invoke the D1 Eyeball Worker, which proxies everything that has to reach the underlying D1 Durable Objects and holds query handling logic — observability, automatic retries, error handling, and other query-specific behavior. Workers KV serves as the edge caching layer for read-heavy metadata and is readable by all D1 Workers.

Users' Workers with a D1 Worker binding send requests straight to the Eyeball Worker, which forwards to the Durable Object backing the database and proxies the response back. That binding and the HTTP query endpoints are the two ways to execute queries.

Operationally, every request falls into one of three categories:

  1. Data plane — any query request; must be as fast as possible and always at the edge.
  2. Control plane — HTTP API operations and internal flows triggered by non-query requests; may depend on central single-region components.
  3. Background asynchronous plane — analytical and billing workflows; eventually consistent and location-agnostic.

Creating a database close to the user

Multi-tenant platforms and agent-driven development need database instances created quickly, cheaply, and reliably. A Wrangler command like the following is the typical entry point:

% time wrangler d1 create "demo-01"

✅ Successfully created DB 'demo-01' in region WEUR
...
0.56s user 0.13s system 42% cpu 1.614 total

Before a new database can accept queries, four things have to happen: the account's free tier or account limits are checked; a unique database UUID, a unique Durable Object ID via newUniqueId(), and the underlying Durable Object are generated and initialized with the account and database information; the database's metadata and its mapping to the Durable Object ID are persisted as one row in Postgres; and the mapping from database ID to Durable Object ID is written to Workers KV for the data plane to use.

The Edge API Worker handler runs in the nearest Cloudflare location. It queries Postgres read replicas over Hyperdrive with caching enabled (cache TTL of a few seconds) to count the account's non-deleted databases. Control plane endpoints that need multiple Postgres round-trips — including "create database" — are forwarded to data centers near the Postgres cluster, because a far round-trip costs hundreds of milliseconds while a nearby one costs a few milliseconds. Endpoints needing at most one Postgres query skip this, since Hyperdrive already routes efficiently over Cloudflare's network.

Creating a database without options places it near the request location. An explicit location hint or jurisdiction — the same options Durable Objects support — is passed straight through to the Durable Object initialization. For the default case, step 4 forwards execution to a data center close to the original request and invokes the Durable Object from there, so the object ends up near the user. The same routing effect can be achieved with explicit Worker placement hints plus Service Bindings at runtime.

Creation is an awkward case for placement: it queries Postgres at least twice but also makes a long-distance call to a Durable Object that varies per request, so either side of the world is suboptimal somewhere. The design leans toward forwarding the initial request near Postgres.

Next, the Worker generates the database UUID and a new unique Durable Object ID with newUniqueId(), then invokes the init(options) RPC function on the new Durable Object's stub. The object persists that information in durable storage, so every later request carries the metadata for its linked database. Once initialization returns, the Edge API Worker writes the database metadata to the central Postgres primary over Hyperdrive, one row per database.

Steps 5 through 7 run inside a retry loop: if Durable Object initialization or the Postgres write fails, the flow retries with different IDs. The Postgres write is the creation checkpoint — the database counts as created from that point even if the final response never reaches the user, and it can immediately participate in control plane operations and be bound to a Worker to serve queries.

The confirmation RPC to the Durable Object that follows runs in waitUntil() and may fail without harm. It tells the object the database is active, and the object updates Workers KV with the database metadata so subsequent queries read it from the Workers KV cache instead of Postgres.

A natural question is why the whole creation flow isn't one Postgres transaction. It wouldn't scale: connections would be held open while the Durable Object initializes, possibly on the other side of the world.

Self-correcting through periodic heartbeats

Each D1 Durable Object heartbeats the control plane on a 30-second interval, plus a per-database jitter. The response carries the latest database state — active or deleted — along with account limits. Both the interval and the heartbeat itself can be tuned per database or per account, including disabling it outright.

Because heartbeats tolerate slightly stale data, they can be served from any of Hyperdrive's caching layers or from a Postgres read replica. That tolerance is what lets the Durable Object converge on Postgres, the source of truth, for database creation and deletion. Heartbeats double as cleanup for orphaned objects.

Why creation retries get fresh IDs

In the creation flow, each retry uses a new database UUID and a new Durable Object ID. Reusing a UUID with a fresh DO ID would leave multiple Durable Objects believing they own the same database — a state that is hard to debug. Ordering matters too: the Durable Object must be initialized before the database mapping is persisted to Postgres, or queries could reach a DO with uninitialized state and be rejected.

A failed write illustrates how the system repairs itself:

  1. The full creation sequence runs through initialization of Durable Object A, then the Postgres write fails.
  2. The Worker retries and initializes a new DO with a different ID, B, and this time the Postgres write succeeds.
  3. Two Durable Objects are now running for the request, but only one is valid.
  4. Queries can only reach DO B, since B is the only ID Postgres holds.
  5. Within 30 seconds, DO A's next heartbeat returns a response saying its database does not exist, and DO A marks itself deleted.

Deletion follows the same pattern in reverse.

Keeping Workers KV honest

The Workers KV update at the end of the creation flow is best-effort and may fail harmlessly. KV is eventually consistent and offers no consistent read, so the Durable Object serves as the serialization point for writes to its items; the next heartbeat rewrites the item with the correct value. Deletion mirrors this: the DO owns writing its deleted state, best-effort during the operation, with the heartbeat as the backstop. The DO also keeps the active-or-deleted state in its own storage, so it can accept or reject queries immediately even when stale KV data lets a request through.

Heartbeats are, in short, the fallback path to consistency whenever an operation must write to an external datastore and update the Durable Object at the same time.

Routing queries around the central database

Query endpoints follow one rule: never touch Postgres or any other single-region centralized component. Each HTTP query hits an Edge API Worker near the user's location, then goes directly to the D1 Durable Object through the Eyeball Worker.

The Edge API Worker handles HTTP API authentication, resource-based authorization, account- and database-level rate limiting, request validation, entitlement checks, and the other API-layer functions before any business logic runs. The Eyeball Worker skips those checks beyond its own dedicated rate limiting, because a D1 Binding is a Service Binding to the D1 Eyeball Worker. The binding encodes authorization for one specific D1 database, which the Eyeball Worker validates before allowing the request through.

The path of a query over the HTTP API:

// Edge API Worker
const parsed = parseQueries(request);

const cachedDB = await withRetries(getCachedDatabase(parsed.dbUuid));

const response = await env.EYEBALL_WORKER.query({
  db: cachedDB,
  queries: parsed.queries,
});

return response;
  1. The user sends a query request over the HTTP API.
  2. The Edge API Worker attempts to load the D1 UUID-to-metadata mapping from Workers KV.
  3. If the item is missing — a failed write as described above, or an availability problem — the mapping is read from Hyperdrive against the Postgres replicas. Hyperdrive checks its cache in the caller's location, then its cache near the Postgres cluster, and only on a double miss queries the replicas.
  4. With the mapping in hand, the Edge API Worker calls the Eyeball Worker over Worker Service Bindings, passing the relevant IDs.
  5. The Eyeball Worker validates and forwards the query to the D1 Durable Object.
  6. The Durable Object executes the SQL and returns the result back through the Eyeball Worker and the Edge API Worker to the user.

Using newUniqueId() at creation rather than addressing Durable Objects by name exists precisely so steps 2 and 3 aren't needed. Name-based addressing would require a lookup round-trip to a central name-to-ID mapping on every query, and the DO ID alone is insufficient — other metadata is needed to select the right Durable Object Namespace in the Eyeball Worker. One lookup would become two. Name addressing remains the right default for most workloads and removes the need to map your own IDs to a DO ID; unique DO IDs are for cases that require them.

Workers KV exists alongside Hyperdrive caching because KV is a durable datastore, not merely a cache. If an item falls out of KV's edge cache, the read is served from KV's own storage instead of Postgres, keeping Postgres out of the data plane. A catastrophic cache purge with a million databases fetching routing information at once would produce metastable failures.

Queries originating from a user Worker with a D1 Binding skip the UUID mapping lookup entirely: the binding carries that information internally, so requests go straight to the Eyeball Worker and the Durable Object. The query execution internals and the Durable Object implementation are covered in the D1 read replication blog post. Next, the asynchronous processes that support D1.

Storage limits: from edge events to account-level enforcement

D1 enforces a 5 GB per-account storage limit on the Free plan and 1 TB on Workers Paid, with the paid ceiling raisable on request. Enforcement systems of this kind tolerate eventual consistency, so D1 treats them as asynchronous work rather than a synchronous part of query execution.

Queues in front of the control plane

Storage usage updates originate in the Durable Objects, which publish them to a Kafka-like queue for batching and asynchronous processing. Processing every update synchronously would offer no benefit; what matters is that a load spike cannot overload Postgres and degrade D1 for other users. Consumer count is capped, which caps the write load on the central regional Postgres store.

The consumers run in Cloudflare's central data centers ("Core"), while the Durable Objects themselves are distributed across edge locations worldwide. Usage is aggregated per edge server before it enters the internal queue system, smoothing and batching what eventually reaches Postgres.

Pre-aggregating per-account totals

For each database, the queue consumer writes the latest reported size into a Postgres table. Summing those rows at enforcement time is not viable: SELECT sum(storage) FROM database_stats WHERE account_id = ? scales linearly with the number of databases on the account, which breaks down for accounts with millions of D1 databases, such as the agentic coding platforms built on D1.

Enforcement instead needs a direct lookup from account ID to total storage, produced with incremental view maintenance. Call the lookup table account_database_stats. Having the queue consumer update both it and database_stats on every incoming update would create heavy write lock contention on a single row for large accounts, so a third table, account_database_stats_changes, serves as a change log with multiple entries permitted per account. Aggregation into account_database_stats happens periodically:

  • The queue consumer writes usage into database_stats for a specific database.
  • A trigger computes the storage difference for that database and records the delta as a row in account_database_stats_changes.
  • A scheduled job runs every second to drain account_database_stats_changes and apply the summed deltas to account_database_stats.

The scheduled job's query is:

WITH updates AS MATERIALIZED (
  DELETE FROM account_database_stats_changes
  RETURNING *
),
changes AS MATERIALIZED (
  SELECT
    account_id,
    SUM(storage_change) AS storage
  FROM updates
  GROUP BY account_id
)
INSERT INTO account_database_stats (account_id, storage)
SELECT
  account_id,
  storage
FROM changes
ON CONFLICT (account_id) DO UPDATE
SET storage = account_database_stats.storage + EXCLUDED.storage;

A Common Table Expression performing DELETE ... RETURNING atomically consumes the pending changes and hands the deleted rows to the next CTE; Postgres executes a data-modifying CTE exactly once and to completion. The query then groups the changes by account and applies a single aggregated update per account, which is what eliminates write contention on individual rows.

The result is eventually consistent rather than true incremental view maintenance of the kind some databases offer first-class. That is acceptable here, because the updates already arrive through a queue, so staleness is baked into the design from the start. Many other Cloudflare teams use the same schema pattern, and it keeps common operations cheap as accounts scale to millions of resources.

Serving limit checks from cache

Each D1 Durable Object fetches its account's usage on the periodic heartbeat. The query SELECT storage FROM account_database_stats WHERE account_id = ? is compared against the account's entitlement and tier limits; a total over the limit blocks all writes that would increase database size.

Reads go to Postgres replicas through Hyperdrive, using its built-in cache. The cache key covers the query text and all parameter values, and the default entry lifetime is 60 seconds, configurable for all queries. Since the account ID is the only parameter, every database on an account shares one cached value.

Queries that need fresher values add a dynamic_ttl parameter, which produces a different cache key every X seconds and causes a fresh read sooner than the default would.

-- $2 = Math.floor(Date.now() / (X * 1000))
SELECT storage, $2 as dynamic_ttl
FROM account_database_stats WHERE account_id = $1

Parameterizing by both account and database ID would answer the tighter question — how much storage this database may use before enforcement begins — and would start enforcement earlier. It would also pull the database ID into the Hyperdrive cache key, wrecking hit rates. With an account-based key, nearly all heartbeats for a large account are served from the same cached value and never touch Postgres. Enforcement reacts to usage changes a little more slowly as a result.

A separate process periodically snapshots storage usage from Postgres into a ClickHouse cluster for a historical view over time. Those ClickHouse events are also the source of truth for storage billing.

Free-tier row limits and the Limiter

Daily execution limits for free-tier customers follow a different pattern. Whenever an SQL query executes in a Durable Object, the rows it reads and writes (including index updates) are counted; on completion, the Durable Object publishes a usage event that lands in a second ClickHouse cluster. These events drive row-based usage billing.

A central component, "Limiter", runs in the Core data centers and periodically reads the day's usage events from ClickHouse:

SELECT account_id, sum(rows_read) as reads, sum(rows_written) as writes
FROM requests
WHERE timestamp > toStartOfDay(now())
GROUP BY account_id HAVING reads > ? or writes > ?

ClickHouse handles this workload better than Postgres, and routing it there keeps analytical queries off the Postgres cluster.

For every account over the limit, the Limiter checks entitlements and tier, since this limit applies only to free-tier users. Over-limit accounts get an email, deduplicated across the day, and a key is set in Quicksilver, Cloudflare's globally distributed key-value store with sub-second replication, expiring at the end of the day. If the account upgrades to Workers Paid, a message on a queue clears that key. Each D1 Durable Object checks Quicksilver periodically at the edge; on finding its account limited, it rejects query execution until the limit lifts. Developers on the platform can use Workers KV Instant, which is built on Quicksilver.

No part of this flow touches the regional Postgres. The pattern is common across the company, though it differs from storage limit enforcement, and other limits may migrate to it in the future.

Conclusion

D1's architecture separates control plane concerns from hot-path query execution. The decisions follow consistent principles: avoid central regional bottlenecks, lean on caching, read replicas and eventual consistency to scale read throughput, and pick the appropriate data store for each job — Postgres for transactional metadata, ClickHouse for analytics, Workers KV for globally replicated routing data.

The patterns and trade-offs described here are the ones actually used in D1 and can serve as a foundation for applications built on Workers and the Developer Platform. Further detail is in the D1 documentation, and the D1 Discord channel is the place to share what you build.

Thanks go to Cloudflare's internal Databases and Hyperdrive teams for finding problematic usage patterns and proposing improvements.