AI Gateway: A Case Study in Network-Bound Compute
AI Gateway is a Node.js service that provides a unified interface to hundreds of AI models, processing billions of tokens daily. At its core, the service is a standard Vercel project running on Fluid compute, Vercel's next-generation runtime designed for highly concurrent, network-bound workloads.
The economics of AI Gateway illustrate a key limitation of traditional serverless pricing. In its first month of general availability, the service accumulated about 16,000 total runtime hours, yet only 1,200 of those hours involved actual CPU work. The remaining 14,800 hours were spent waiting for AI providers to respond. Traditional serverless platforms bill for wall-clock time, meaning every millisecond the function is alive incurs CPU costs. With Fluid's Active CPU Pricing, CPU rates apply only when the CPU is actively executing. The rest of the time, when the service is blocked on a provider response, it pays a reduced memory-only rate. For AI Gateway, this translates to paying CPU rates for under 8% of runtime rather than 100%.
Inside the Request Path
AI Gateway is a Node.js application built with Next.js, though any backend framework serving REST APIs would work. It runs across Vercel's distributed infrastructure in multiple AWS regions, using the same networking, global regions, and compute model available to all Vercel customers. The application holds no special privileges or custom infrastructure.
When a request enters AI Gateway, it first hits the Vercel global delivery network, a globally distributed system combining Anycast routing, Points of Presence (PoPs), and private backbone connectivity. The network continuously evaluates endpoint health and congestion, routing requests to the optimal PoP based on proximity and live telemetry. Traffic is then handed off to the nearest compute-capable region.
At the application layer, the function authenticates using OIDC tokens for Vercel-hosted apps or API keys for external integrations. It verifies quotas through in-region Redis, prepares the payload for the target AI provider, and streams responses back through the same low-latency network path. Every hop, from client ingress to provider response, stays within Vercel managed boundaries, avoiding unpredictable public internet routes. The architecture keeps round-trip times in single-digit milliseconds for most customers.
How Fluid Differs from Serverless
Traditional serverless models require a separate instance for every invocation. Even pre-warmed instances start with no memory or state. Fluid changes this model by reusing instances not just after they finish running, but while they are still executing, through in-function concurrency. When one invocation pauses to wait for a provider response, another can execute immediately within the same instance.
This concurrency model allows in-memory data, open sockets, and runtime caches to persist throughout an instance's lifecycle. Fluid instances can store small, short-lived caches of frequently accessed data, such as provider routes, credentials, or quota snapshots. This reduces redundant Redis lookups and minimizes latency on the hot path. When traffic spikes, Fluid scales instantly; when traffic quiets, instances retire gracefully. Infrastructure remains elastic like serverless but performs like an always-warm server.
CPU utilization stays high while costs stay low because Active CPU Pricing charges only for the milliseconds when code is actively running.
State, Monitoring, and Failover
AI Gateway uses Redis for global consistency and quota tracking, while Fluid handles ephemeral in-memory caching for local speed. Frequently accessed credentials, provider metadata, and quota counters live in instance memory for sub-millisecond access. Redis values are refreshed asynchronously in the background, with writes and usage increments batched and written back after responses complete. Each Vercel region maintains its own Redis cluster, keeping quota verification and usage updates local and predictable.
Monitoring relies on two complementary systems. Health checks continuously measure core metrics like error rates, time to first token (TTFT), and throughput in tokens per second. In parallel, every Fluid instance maintains real-time counters tracking active invocations, memory utilization, and provider latency. The monitoring system compares in-memory telemetry against global health checks, using the data to adjust routing, scale instances, or shift traffic between regions automatically. If one provider region starts returning slower responses, AI Gateway reroutes new requests to a healthier provider without human intervention.
AI Gateway supports routing requests across multiple providers for the same model and can fall back to other models in the event of an error, context size mismatch, or other incompatibility. For example, Claude Sonnet 4 is available through Anthropic, Amazon Bedrock, and Google Vertex AI. Developers control which providers are used and in what order. If the primary provider is unavailable, AI Gateway automatically retries with the next one. Every response includes detailed metadata describing which provider served the request, any fallback attempts, latency, and total token cost. Vercel Observability provides real-time visibility into request volume, spend, and performance.
Matching Pricing to Workload
AI Gateway spends most of its time waiting, not computing. Routing requests, verifying credentials, and checking quotas takes milliseconds. Waiting for a provider to stream back a response takes seconds. This is a network-bound workload, not a compute-intensive one. Active CPU Pricing matches this pattern: during waiting periods, you pay only for memory provisioning, not full CPU rates. For workloads like AI Gateway, where most time is spent waiting on network responses, this eliminates unnecessary costs.
The same architecture that powers AI Gateway is available to every developer building AI-powered applications on Vercel. Fluid's concurrency model and Vercel's distributed network demonstrate what modern infrastructure looks like when serverless evolves beyond simple functions — instant and elastic, but also intelligent, efficient, and self-optimizing.
