From 30 minutes to indefinite: How AI Gateway rearchitected log storage
AI Gateway, which has proxied over 2 billion requests since its September 2023 launch, initially faced a hard constraint: logs were only retained for 30 minutes. That was a real problem for developers who needed to analyze long-term patterns, ensure compliance, or debug issues over extended periods. The engineering team had to rethink the storage architecture entirely to scale from transient records to indefinite, queryable history.
The service, built on Cloudflare Workers, lets developers route inference requests to multiple providers and models through a single gateway. Each request and response is logged so customers can inspect cost, duration, token counts, and provide feedback. Requests surface within 15 seconds. But the original architecture, which relied on a backend worker for log storage and background tasks, could not keep up with the data volume.
The evolution: from D1 to R2 to Durable Objects
The first implementation stored both request metadata and bodies in a D1 database. As customer engagement grew, D1 filled rapidly and retention dropped to just 30 minutes. Schema optimization stretched that to one hour, but diminishing returns quickly appeared due to the size of request bodies. The team then moved request bodies to R2 storage, which reduced the load on D1 and allowed retention to be extended to 24 hours. In this dual-system design, D1 became a log index for search and filtering, while details and downloads were proxied from R2.
The next phase leveraged the Durable Objects team's beta for SQLite support. The team initially sharded logs by account ID, but testing revealed a cap of 10 million logs per Durable Object. Refining the approach, logs were sharded by both account ID and gateway name. This raised the ceiling to 10 million logs per gateway, and with the default limit of 10 gateways per account, potential storage per account became 100 million logs.
This sharding strategy brought operational benefits. Deleting a gateway simply removes the corresponding Durable Object. High-volume customers are isolated; if one customer's heavy usage slows log insertion, it only impacts their own Durable Object, not others.
The revised flow keeps the Gateway Worker close to the user on Cloudflare's network, positioning the Durable Object nearby for fast log insertion and queries.
Introducing the Account Manager
As users grew, managing thousands of Durable Objects became complex. New customers join continuously, and the team needed to track each Durable Object, enforce the 10-gateway limit, and manage storage for free users. The solution was a new Durable Object called the Account Manager.
Before a Gateway commits a log to permanent storage, it consults the Account Manager. That component checks the user's current usage and entitlements, using its SQLite database to verify total rows and service level. If checks pass, the Gateway is signaled to proceed with insertion. All validation happens in the background so user experience remains uninterrupted.
To stay current, the Account Manager receives periodic updates from each Gateway's Durable Object. After every 1,000 inference requests, the Gateway sends its total row count to the Account Manager, which updates its records. The Account Manager also tracks whether each account is free or paid, how many gateways a user can create, and the log storage capacity per gateway.
This system maintains integrity and ensures fair usage across all AI Gateway users.
Scaling for AI evaluations
AI Gateway is now entering the open beta of comprehensive AI evaluations, focused on Human-in-the-Loop feedback. This feature lets users create datasets from application logs to score model performance, speed, and cost-effectiveness, with a focus on LLMs and automated scoring.
Reliable, persistent log storage is a prerequisite. Developers need to store up to 100 million logs total, across multiple gateways, with some individual log entries exceeding 50 MB. Since the current limit is 10 million logs per gateway, the team is working on sharded Durable Objects that would allow multiple Durable Objects per gateway. This will enable significantly larger log volumes, providing richer data for evaluations, all through AI Gateway.
Improved Universal Endpoint: Retries and Conditional Logic
Cloudflare is extending its AI Gateway with an enhanced Universal Endpoint designed to add automatic retry capabilities and content-aware decision-making to the existing fallback mechanism. The current system only routes to an alternative provider when the primary one fails; the improved version aims to reattempt failed requests before falling back, reducing the impact of transient errors on overall reliability.
The new flow introduces a retry step that precedes fallback logic. In a typical sequence, a first request is sent to a given provider—for example, OpenAI—with a “retry” option enabled. If that request fails due to a temporary issue, the system retries it rather than immediately treating the attempt as a failure. Only after retries are exhausted does the fallback to another provider trigger. This layered approach allows the gateway to handle multi-step workflows where each stage may involve a different provider, keeping the overall job moving forward even when individual calls encounter issues.
Beyond retries, the improved Universal Endpoint will allow inspection of both requests and responses at runtime. Developers can define conditional workflows where the gateway makes decisions based on the actual content of a response, not just its status. This opens the door to branching logic that adapts to what a model returns, enabling more sophisticated orchestration of AI calls without moving the state management into application code.
The combination of retry strategies, fallback logic, and response-driven branching is intended to make provider failures and transient network errors less disruptive. Requests that would previously have required manual intervention or a full re-issue can now be handled inside the gateway itself.
Developers interested in using AI Gateway can refer to the official developer documentation, and questions can be directed to the Cloudflare Discord channel.
