Why Cloudflare built Omni
Workers AI hosts a large catalog of models, but traffic isn't evenly distributed across them. Quiet models were occupying valuable GPU memory on edge nodes, running idle while scarce accelerators sat underutilized. Cloudflare's answer is Omni, an internal platform that runs multiple AI models on a single machine and GPU using lightweight isolation techniques rather than dedicating a GPU (or even a container) to each model.
The core techniques are:
- Spawning and supervising many model instances from a single scheduler
- Process isolation with cgroups and Python virtual environments instead of full containers
- Per-model file system isolation for dependency management
- GPU memory over-commitment, letting more models share a physical GPU than would normally fit
With models packed more densely per node, GPU availability improves across Cloudflare's network, per-request latency drops due to better locality, and power draw from idle GPUs is reduced.
A scheduler per node, a process per model
When an inference request hits Workers AI, model configuration is loaded from Workers KV and a routing layer forwards it to the nearest Omni instance with capacity. Batch API requests are sent to idle instances, which typically means routing to nodes where it's currently nighttime. From there, Omni runs pre- and post-processing specific to the model and then hands the request to the model itself.
The lifecycle of every model process is managed by a single scheduler per Omni deployment. The scheduler provisions new instances as traffic grows, which means downloading model weights, Python code, and dependencies on demand. It routes incoming requests to the correct process, spreads load across GPUs, supervises running processes, restarts failed ones, rolls out model version updates, and collects metrics and logs for billing.
Inference itself runs in a separate child process supervised by the scheduler. The scheduler and child processes communicate over Inter-Process Communication (IPC). Requests are usually buffered in the scheduler so features like prompt templating and tool calling can be applied first. For large binary request bodies, the scheduler hands the underlying TCP connection directly to the child process.
Isolation without containers
Running each model in its own container wastes memory and makes it harder to colocate models with different dependency sets. Omni instead runs many models inside a single container (or on bare metal for single-model deployments), isolating each model via process namespaces, cgroups, and Python virtual environments.
One problem emerged: Python libraries commonly use psutil to decide how much CPU memory to pre-allocate, but psutil reads /proc/meminfo, which on a standard system reflects host memory rather than a cgroup's limit. Python's memory allocator also ignores cgroup limits, so a model could get OOM-killed despite the container having free memory.
Omni solves this by mounting a virtual /proc/meminfo using FUSE, exposing memory usage and limits for just that model instead of the whole host. The example below is a live Omni instance: the model sees a 7 GiB limit inside a 15 GiB container.
# Enter the mount (file system) namespace of a child process
$ nsenter -t 8 -m
$ mount
...
none /proc/meminfo fuse ...
$ cat /proc/meminfo
MemTotal: 7340032 kB
MemFree: 7316388 kB
MemAvailable: 7316388 kBIf that model exceeds its 7 GiB allocation it is OOM-killed and restarted by the scheduler — isolation limits blast radius to a single process. Python and system dependencies are managed through virtual environments created by uv, which shares cached packages across environments via symbolic links. Separation of processes also means independent CUDA contexts, an aid to error recovery.
Over-committing GPU memory
Not all models generate enough traffic to saturate a GPU, so Omni packs more of them onto each device by over-committing memory. With 10 GiB of physical GPU memory, for example, Omni can run two models each expecting 10 GiB — in one production configuration, 13 models run against roughly 400% of physical GPU memory, saving the need for 4 additional GPUs. Trade-offs come in the form of data movement: if an inference needs weights that were swapped to host memory, those must be transferred back over PCIe.
Omni accomplishes over-commitment by injecting a CUDA stub library that intercepts memory allocation calls (cudaMalloc*/cuMalloc*) and forces allocations into CUDA unified memory mode. Unified memory gives CPU and GPU a shared address space, with pages migrated to whichever device needs them:
In practice the scheduler shuffles model weights between GPU and host memory as traffic patterns shift:
- Models A and B are resident on the GPU, while model C's weights wait in CPU memory.
- A request arrives for C. A and B are swapped out, and C is swapped into GPU memory.
- A request for B arrives. C is evicted, at least partially, and B is swapped back in.
- A request for A arrives. A is swapped back in, and C is fully evicted.
Swap latency depends on model size and bus bandwidth. With PCIe 4.0's 32 GB/sec, a 5 GiB model takes roughly 156 ms to move back into GPU memory. Smaller models get proportionally faster cold starts. Over-commitment never lets a greedy model take over the device: Omni also overrides cudaMemGetInfo and cuMemGetInfo to expose only a per-model slice of GPU memory to each child process, preventing a single model from pre-allocating the whole GPU for itself.
The model handler interface
Workers AI models run on several inference engines — vLLM, plain Python, and Cloudflare's own Infire. Each model must still expose Workers AI features such as batching and function calling, so Omni acts as an integration layer between the routing/scheduling systems and whichever engine the model uses.
Adding a model to Workers AI means writing a request handler function, mirroring the JavaScript Worker model:
from omni import Response
import cowsay
def handle_request(request, context):
try:
json = request.body.json
text = json["text"]
except Exception as err:
return Response.error(...)
return cowsay.get_output_string('cow', text)Python dependencies are declared in a requirements.txt, installed at model startup from an internal registry that mirrors the public one.
cowsay==6.1An injected Python package called omni provides APIs for interacting with the request, Workers AI platform, building responses, and error handling. Handlers support async functions, return rich types such as pydantic objects, and are usable outside Omni for unit testing.
from omni import Context, Request
from model import handle_request
def test_basic():
ctx = Context.inactive()
req = Request(json={"text": "my dog is cooler than you!"})
out = handle_request(req, ctx)
assert out == """ __________________________
| my dog is cooler than you! |
==========================
\\
\\
^__^
(oo)\\_______
(__)\\ )\\/\\
||----w |
|| ||"""Omni is currently in production for a subset of Workers AI models, with more added every week. The platform's ability to start and stop models quickly, keep their dependencies isolated, and share GPUs between multiple workloads lowers the cost per inference and lets Cloudflare add new models to the catalog faster.
