Serverless That Behaves Like a Server

Vercel has announced a public beta for in-function concurrency in Vercel Functions, a feature designed to bring the efficiency of traditional servers to serverless compute. The company reports that early customers have seen a 20-50% reduction in compute usage and associated costs, with no corresponding impact on latency.

This new capability is aimed squarely at interactive workloads—server-rendered pages, APIs, and AI applications—where individual requests often leave compute idle while waiting for backend responses. Vercel Functions retains its existing Node.js support, including V8 bytecode caching for faster cold starts and instance pre-warming for production traffic.

Moving Past the One-to-One Model

The original "functions as a service" paradigm, popularized by AWS Lambda, established a simple but limiting rule: one function instance per invocation. This model works well for batch jobs but is inefficient for dynamic, request-driven workloads. A single user request might hold a function instance hostage while it waits seconds for an AI model or another backend service to respond.

In-function concurrency breaks that mapping. Instead of tying each invocation to a dedicated instance, Vercel now allows a single instance to handle multiple calls simultaneously. The idle time spent waiting on I/O becomes available for processing other requests.

The Plumbing Behind the Feature

The path to concurrency required several infrastructure changes. In 2022, Vercel released Next.js 13 with streaming support for React server-side rendering. At the time, Lambda's lack of streaming support forced a workaround: functions would open a secure TCP socket back to Vercel's function invocation service and stream responses through it, bypassing Lambda's limitations.

This bidirectional stream turned out to be more than a workaround. It opened the door to sending additional work to functions mid-invocation. To make that viable, Vercel rewrote its function runtime in Rust. The rewrite delivered performance improvements and better debuggability, and it now serves as a stable orchestrator that keeps workloads running even if an individual invocation runs into trouble.

The resulting architecture routes incoming traffic through a function load balancer to the invocation service. That service manages connections ranging from a single instance up to 100,000 instances per function, routing each request to an instance that already has in-flight invocations when one is available.

Measuring the Efficiency Gains

The efficiency improvement comes from utilising compute that would otherwise be idle. In the old model, two requests arriving at the same time require two separate instances and twice the compute time. With concurrency, a single instance can take the first request and, while it waits for a backend response, pick up the second.

Consider a request that takes 100ms total—50ms of computation and 50ms waiting on a backend. Under the old model, two requests consume 200ms of compute. With in-function concurrency, the same two requests can be handled by one invocation, reducing actual compute time to roughly 100ms. That translates to fewer gigabyte-hours billed, with no code changes required.

The magnitude of the gains depends on traffic volume and how much time requests spend waiting on external services. Concurrency is most effective when there are enough in-flight requests to fill the idle gaps.

Node.js Finally Running as Intended

Node.js was designed from the ground up for concurrent asynchronous I/O. In practice, though, serverless platforms have rarely given it the chance to do so. Traditional functions-as-a-service products could exploit concurrency only within the context of a single invocation, leaving much of the CPU untouched while a request waited on a database call or an external API.

With in-function concurrency, a single Node.js process can handle multiple invocations at once. When a function stalls waiting for a backend, it can immediately process another request using the available CPU. For IO-heavy workloads, this means compute is continuously utilised rather than sitting idle.

Beta Limitations and Trade-offs

Vercel is being cautious about how fast it pushes concurrency limits. During the public beta, the maximum number of concurrent invocations dispatched to a single instance is intentionally capped, with limits slated to rise gradually as the company monitors the balance between efficiency and latency.

Not every workload benefits equally. For purely CPU-bound tasks, the mechanism can potentially increase latency. Vercel expects its detection of unhelpful concurrency to improve during the beta, ensuring the feature is used only where it makes sense.

There's also a semantic shift to consider: a single Node.js process is now handling multiple invocations concurrently. That's a backward-incompatible change, so the feature is opt-in. Vercel notes that this is how Node.js is typically run in production anyway, and expects most workloads to work under the new semantics without issue.

Still Serverless at Heart

Despite the architectural changes, Vercel stresses that the serverless properties remain intact. Functions still scale automatically, require no maintenance windows, and receive operating system and runtime security updates transparently, with no downtime or user action required.

Verse's Real-World Cost Reduction

One early adopter, Verse, saw traffic surge from hundreds of thousands to over 10 million visitors following the launch of its internet bedroom website. At peak, 15,000 users were online simultaneously, generating a high volume of request traffic and plenty of idle compute time.

With early access to in-function concurrency, Verse cut its GB-hours and reduced costs by over 50%. Aydan Gooneratne, Lead Full Stack Developer at Verse, noted that his team's API endpoints were lightweight and often involved external requests, leaving compute idle. The new feature allowed them to share compute resources between requests, cutting costs by over 50% without writing a single line of new code.

Availability and Observability

In-function concurrency is now available in public beta for all Pro and Enterprise customers. It can be enabled from the Functions tab in project settings. Vercel is also adding a new Observability tab for serverless functions, allowing users to view cost savings alongside key metrics such as latency and time to first byte.