The life of an Ollama prompt - Eli Bendersky's website

Anatomy of an Ollama request

Ollama's convenience hides a fairly involved request pipeline. When you send a prompt via the generate endpoint, it must first retrieve the model, translate the request into something an inference engine can consume, and stream back the response the engine produces. Looking at what happens internally clarifies the distinct layers in that process.

Request entry points

Every interaction with Ollama, regardless of the client, funnels through its REST API. Ollama handles three client styles: raw HTTP requests via tools like curl, Ollama's own client libraries for Go, Python and JavaScript, and provider-agnostic libraries like LangChainGo.

A typical raw call hits the api/generate endpoint:

$ curl http://localhost:11434/api/generate -d '{
  "model": "gemma",
  "prompt": "very briefly, tell me the difference between a comet and a meteor",
  "stream": false
}' | jq .

[...]

{
  "model": "gemma",
  "created_at": "2024-03-04T14:43:51.665311735Z",
  "response": "Sure, here is the difference between a comet and a meteor:

  **Comet:**
  - A celestial object that orbits the Sun in a highly elliptical path.
  - Can be seen as a streak of light in the sky, often with a tail.
  - Comets typically have a visible nucleus, meaning a solid core that
    can be seen from Earth.

  **Meteor:**
  - A streak of hot gas or plasma that appears to move rapidly across the sky.
  - Can be caused by small pieces of rock or dust from space that burn up
    in the atmosphere.
  - Meteors do not have a visible nucleus.",
  "done": true,
  "context":
[...]
}

The service layer is itself a binary running as a background process. When you invoke ollama run gemma on the command line, the binary switches to client mode and sends the same HTTP request to http://localhost:11434/api/generate that you could have issued yourself with curl. Ollama listens on port 11434 by default unless the OLLAMA_HOST environment variable overrides the host or port.

Request handling

The API routes are registered using Gin, with the generate route mapping directly to a handler in the same source file:

r.POST("/api/generate", GenerateHandler)

The handler first parses and validates the model name in the request body. It then has to make sure the model exists locally, and if not, fetch it. Once the model is loaded, the prompt gets passed on for inference.

Model lookup and retrieval

Ollama keeps a local model cache in its data directory (on Linux, typically under /usr/share/ollama/.ollama/models/blobs). If the requested model isn't already in that cache, the service checks the online registry at https://registry.ollama.ai/v2/library/ and downloads it on demand.

The actual files in the registry are in GGUF format, which stores both model metadata (architecture type, layer counts, etc.) and the model weights themselves. Weight formats vary, and quantization is common, especially for models aiming at CPU inference. Regardless of the provenance of a particular model—it may originate from GGUF, Safetensors, or other formats—Ollama's own storage eventually settles on GGUF, and files can run into multiple GiB each.

Behind the curtain: llama.cpp

Ollama itself doesn't perform inference. That job goes to llama.cpp, a C++ implementation of local inference written on top of a separate ML primitives project called ggml. The project started with the original Llama release, hard-coding that architecture, but has since expanded to support many open-sourced models through a dispatch switch based on each model's declared architecture. Calling into llama.cpp from Go requires a compatibility layer, since C++ lacks a stable ABI. Ollama established a glue binding through its ext_server subproject, using cgo to invoke the C++ engine in-process. On the Go side, the generate endpoint calls llm.Predict, which ultimately hands off to llama.cpp's request_completion function defined in its server sample. That server code already accepts JSON input and returns JSON output, so the bridge stays relatively thin.

Standardization as convenience

The smooth experience working with local models comes down to llama.cpp establishing two de facto standards: the GGUF file format and a framework for local inference. Once a model's architecture is implemented in llama.cpp, publishing a variation, a tuned checkpoint, or a newly quantized run is just a matter of emitting the right GGUF file. The same codebase will then execute it on whatever CPU or GPU is available.

On top of that foundation, Ollama contributes the packaging and REST layer: a simple API server that any HTTP-capable tool can invoke, regardless of programming language. For Go developers specifically, Ollama's own client library closely mirrors what the command-line client uses internally, while LangChainGo targets codebases that need a single, provider-neutral interface across several model services.