Why GenAI traffic needed its own lane in Meta's web tier
Meta’s Web Foundation team runs the company’s monolithic web tier on HHVM, the HipHop Virtual Machine. Operating that shared tier means enforcing rules that keep any single workload from starving the others. The most important of those rules is a 30-second request runtime limit. Each request maps to one worker thread, and the thread count is finite; if requests run too long, threads back up and the host stops accepting new work.
Classic web traffic coexists comfortably with that limit. Front-end requests like page renders and GraphQL queries finish in hundreds of milliseconds to a few seconds, so a typical host can sustain roughly 500 queries per second. Those requests are also I/O-bound—a web server spends about two-thirds of its time waiting on I/O and one-third doing CPU work—which is why Hack and its core libraries are built around asyncio-style cooperative multitasking. The design keeps the CPU busy while sockets idle.
GenAI inference flips those assumptions. When a user submits a query to an LLM, the model streams tokens back over seconds or even minutes. The critical path before that stream starts must be as short as possible, but the overall request lifetime is far longer than a typical front-end request, and most of that lifetime is spent waiting on I/O. The old rules simply did not fit.
Rethinking the request runtime rules
Instead of loosening the shared-tier constraints and hoping nothing broke, Web Foundation spun up a dedicated web tenant—a standalone WWW deployment—with configuration tuned to the demands of GenAI work. That isolation opened the door to several changes.
A longer leash on request timeouts
The first change was straightforward: raise the runtime limit for GenAI requests. On an isolated tier, longer-running inference no longer threatens to consume worker threads needed by ordinary production traffic, so requests can finish without tripping a 30-second timeout.
Bigger thread pools
Longer request lifetimes shrink the pool of available worker threads. Because memory on a host is fixed, the engineering team sized thread pools by dividing total memory by the per-request memory limit to derive a ceiling for concurrent requests. The result is roughly 1,000 threads on GenAI hosts, versus a couple of hundred on standard webservers.
JIT cache seeding with Jump-Start
HHVM is a JIT-interpreted language: the first time a function runs, the runtime compiles it to machine code, which adds latency on cold paths. Meta's Jump-Start technique lets a server seed its JIT cache with profiles from an already-warmed host. GenAI hosts pull Jump-Start profiles from the main web tier, which meaningfully reduces compilation overhead even when the code paths don't perfectly overlap.
Pre-flight warm-up requests
Beyond code compilation, HHVM supports executing dummy requests at startup and discarding the results. The purpose is to warm non-code caches: configuration values and service discovery data are normally fetched lazily on first use, then cached. Warm-up requests front-load those fetches so real users never see the latency spike of an initial lookup.
Shadow traffic to stay warm
Meta controls feature rollouts with real-time configuration, which means a Jump-Start profile captured at startup may not cover code paths enabled moments later. To preserve JIT coverage in the steady state, the team added request shadowing. Because gating changes can activate new code paths at any time, shadow requests ensure those paths still make it into the JIT cache before real users hit them.
Collectively, the isolated tenant, boosted thread count, Jump-Start seeding, deliberate warm-up, and shadow traffic delivered a 30% improvement in latency—while keeping GenAI's long-streaming requests from degrading the rest of Meta's web infrastructure.
