LoRA integration and runtime mechanics
Because Workers AI models stay pre-loaded on the GPUs, requests never hit a cold start. The base model is therefore always resident, and LoRA adapters can be loaded and swapped in dynamically as requests arrive. Multiple adapters can be attached to a single base model at the same time, so several different fine-tuned inference requests can be served concurrently.
At inference time only the smaller matrices A, B are needed to change model behavior. Those weights, together with a config file, make up the LoRA adapter. At runtime the original model (W0) and the adapter are summed. Since addition and subtraction are simple operations, adapters can be swapped quickly by adding or subtracting A, B from W0, delivering fine-tuned output with minimal added latency.
The original LoRA paper reports that LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times." Compared with a fully fine-tuned model, it is far cheaper to compute, adds no material inference time, and is smaller and more portable.
Using BYO LoRAs on Workers AI
A LoRA fine-tune produces two files: the custom weights in safetensors format and an adapter config file in JSON. You can train these yourself with the Hugging Face PEFT library together with the Hugging Face AutoTrain LLM library, or run training on services such as Auto Train and Google Colab. Open-source adapters covering many use cases are also available on Hugging Face. Training workloads on the platform itself are planned; for now, adapters must be trained elsewhere, hence the Bring Your Own (BYO) LoRAs name.
The open beta supports LoRAs with Mistral, Llama, and Gemma. LoRA-capable versions of these models are exposed by appending -lora to the model name, and an adapter must have been fine-tuned from one of these base models:
@cf/meta-llama/llama-2-7b-chat-hf-lora@cf/mistral/mistral-7b-instruct-v0.2-lora@cf/google/gemma-2b-it-lora@cf/google/gemma-7b-it-lora
Beta limitations to keep in mind: quantized LoRA models are not yet supported; adapters must be smaller than 100MB and have a maximum rank of 8; and each account can try up to 30 LoRAs during the initial open beta. Setup instructions are in the Developer Docs. Usage remains subject to the Terms of Service, including any model-specific restrictions in the models' license terms.
Background: what fine-tuning changes
Fine-tuning modifies an AI model by continuing to train it on additional data, raising the probability that generations resemble that dataset. Training from scratch is expensive and time-consuming for most use cases, so fine-tuning an existing pre-trained model lets you keep its capabilities while targeting your task. LoRA is a fine-tuning method applicable to various model architectures, not only LLMs. Traditional methods modify or fuse pre-trained weights directly; LoRA keeps the fine-tune weights separate and leaves the pre-trained model unchanged. The results can be more accurate at specific tasks — generating code, adopting a personality, producing images in a given style, or understanding a specific topic.
Models are neural networks whose parameters, made up of weights and biases, act as a mathematical representation of domain knowledge and are usually stored as large matrices. Parameter count defines model size, so llama-2-7b has 7 billion parameters. Trained from scratch, these numbers start random and are adjusted during training until the model reflects the dataset; weights express how much importance each parameter carries. Traditional fine-tuning adjusts all parameters, so a fully fine-tuned model requires serving as many parameters as the original, at high training and inference cost. Because new state-of-the-art models ship regularly, fully fine-tuned models are also costly to train, maintain, and store.
Why low rank works
LoRA instead applies a small number of additional parameters temporarily to the base model to control its behavior. Training them takes far less time and compute than traditional fine-tuning, and the resulting adapter is packaged as a separate model file that plugs into its base model. Fully fine-tuned models can be tens of gigabytes; adapters are usually a few megabytes, which makes them easy to distribute.
In matrix terms, tuning the original weights W0 yields a change ΔW, so the tuned model is W0 + ΔW. Every matrix has a rank (r), the number of linearly independent columns or rows. Low-rank matrices have few "important" rows or columns and can be decomposed into two smaller matrices holding the most important parameters — rank decomposition. In fine-tuning, rank determines how many parameters change from the original model: the higher the rank, the stronger the fine-tune and the more granular the output.
Per the original LoRA paper, when a model is low-rank the matrix representing the weight change is also low-rank, so rank decomposition can be applied to ΔW to produce two smaller matrices A, B with ΔW = BA. The change is then represented by two smaller low-rank matrices — hence Low-Rank Adaptation.
Running many LoRA adapters side by side is a GPU scheduling problem rather than an inference problem. Batched requests against a single base model are straightforward; batched requests that each carry a distinct adapter are not. Workers AI addresses this with the Punica CUDA kernel design plus global cache optimizations, aiming to keep memory use under control without sacrificing latency.
Segmented Gather Matrix-Vector Multiplication
Punica, described in the paper Punica: Multi-Tenant LoRA Serving, is a method for serving several substantially different LoRA adapters over one shared base model, and reports sizable throughput and latency gains over earlier inference approaches. A key reason is that it permits batching across requests that target different adapters.
The centerpiece is a new CUDA kernel, Segmented Gather Matrix-Vector Multiplication (SGMV). With SGMV, the GPU holds only a single copy of the pre-trained model while still serving distinct LoRA models. Requests for different adapters are batched together and their feature-weight multiplications parallelized; requests sharing an adapter are then grouped to raise operational intensity.
Memory layout and adapter loading
At startup the GPU loads the base model and reserves the bulk of its memory for the KV Cache. The A and B matrices that make up a LoRA adapter are not preloaded: they are pulled on demand from remote storage — Cloudflare's cache or R2 — when an incoming request calls for them. That fetch costs only milliseconds, so adapters can be retrieved and served with little effect on inference performance, and adapters that are requested often stay cached for the fastest path.
Once an adapter is local, the limiting factor for making it usable is PCIe bandwidth. Since every request may need its own adapter, downloads and memory copies have to run asynchronously. The Punica scheduler handles this by batching only those requests whose LoRA weights are already resident in GPU memory; the rest wait in a queue until their weights arrive and they can join a batch efficiently.
Continuous batching
Managing the KV cache and batching carefully makes large multi-tenant LoRA workloads feasible. Continuous batching adds another gain: conventional batching holds every request to the same adapter until all have hit their stopping condition, whereas continuous batching lets an individual request leave a batch early instead of waiting on the longest-running one.
Because the LLMs deployed on Cloudflare's network are available globally, the adapters need to be too. Remote model files cached at Cloudflare's edge are planned to cut inference latency further.
Where fine-tuning on Workers AI goes next
LoRA adapter support is a step toward fine-tunes as a first-class capability on the platform. Beyond the LLM fine-tunes offered today, the plan covers additional models and more task types, image generation among them.
The longer-term goal is for Workers AI to host the fine-tuning process itself, not just inference: running the training job and fully fine-tuned models directly on the platform. That would let organizations make models more specific to their tasks, with finer granularity and detail.
AI Gateway fits into this by logging prompts and responses that developers can later use to fine-tune with production data. The intended end state is one-click fine-tuning: log data from AI Gateway retrains a model on Cloudflare, and the result is redeployed on Workers AI for inference. Models could then be personalized down to the per-user level, and a smaller, more optimized fine-tuned model would reduce inference time and cost — all inside the Developer Platform.
The BYO LoRA open beta is available now; details are in the Developer Docs, and feedback is welcome on Discord.



