ML in Go: from hosted APIs to Python sidecars
Go developers who want to add machine learning to their applications have more options than ever. The path from a fully hosted commercial LLM to a completely custom model is a spectrum, and where you land on that spectrum determines how much — if any — Python you'll need to write.
The easiest path: hosted LLM services
Commercial LLMs like ChatGPT, Gemini, and Claude are exposed as language-agnostic REST APIs. Go is as well supported as any other language here, with official client libraries and third-party abstractions like langchaingo available. As the official Go blog noted earlier this year, Go's network-native nature makes it uniquely suited for LLM-powered applications, which are fundamentally about juggling concurrent requests to network services.
Running open models locally
Open models like Gemma, Llama, and Mistral have become surprisingly capable. For applications where cost or privacy rules out commercial services, running these locally is a compelling alternative. A few factors have made this practical:
- Standardized model formats like GGUF from
llama.cpp, Hugging Face's safetensors, and ONNX - OSS tools that expose local models via familiar REST APIs
Ollama is the most widely known of these tools. Configured through a Modelfile, it supports setting model parameters, system prompts, and loading fine-tuned GGUF models. For cloud environments, GCP's Cloud Run integration is an option. A more recent alternative, Llamafile, distributes an entire model as a single portable binary with REST APIs, another build-and-forget approach.
If a locally-running open LLM fits the bill, running Ollama or Llamafile and talking to it via its REST API is a practical, low-effort solution.
For deeper customization, a more general architectural pattern is needed — one that doesn't necessarily involve a heavyweight model server.
Beyond existing models: the sidecar pattern
Sometimes an existing open model won't do. Training a custom model means working with TensorFlow, JAX, or PyTorch, none of which have real non-Python alternatives. cgo won't help if there's no C API either.
A general solution is the sidecar pattern: wrap the functionality in a server interface and run it as a separate process. While the term comes from containerized Kubernetes deployments, the pattern applies to any architecture where functionality is isolated across processes. The benefits include isolation, security, and language independence. The earlier Ollama discussion is already an implicit example — but with a dedicated server. For truly custom models, you build the sidecar yourself.
The approach is straightforward: wrap your Python ML inference code in a lightweight HTTP server, expose a minimal REST interface, and let your Go application talk to it over localhost.
The fine-tuned approach: a JAX-based Gemma sidecar
Consider the case where you have a fine-tuned model, trained in Python with a framework like JAX. A simple Flask server around the inference code is enough to make it usable from Go. Here's what it takes:
Using the official Gemma repository as a base, a sampler is instantiated with model weights and a tokenizer vocabulary downloaded from Kaggle. The web server, launched with gunicorn, exposes two routes: prompt for sending a textual prompt and receiving generated text in a JSON response, and echo for testing and benchmarking.
The key point is that the concrete technologies are replaceable. PyTorch could replace JAX; a non-LLM model could replace Gemma; any HTTP server framework could be swapped in. The pattern stays the same.
The Python code totals under 100 lines, most of it pieced together from tutorials. The application's business logic remains in Go.
Is the IPC overhead a concern?
For a large model like Gemma, where a prompt takes seconds to process on a GPU, the inter-process communication is irrelevant. A simple echo benchmark shows a round-trip JSON request from Go to the Python server averaging about 0.35 ms.
But that number won't hold for models that are small and fast. When processing takes milliseconds rather than seconds, a 0.35 ms overhead per request becomes a meaningful fraction of the total cost. That scenario requires a different, lower-overhead solution.
A Lower-Latency Sidecar: Image Classification Over a Unix Socket
The final example in this post intentionally diverges from the LLM pattern used so far. Instead of a large language model, it trains a small convolutional neural network to classify CIFAR-10 images, and instead of HTTP+REST, the Python sidecar communicates over a Unix domain socket with a custom protocol. The goal is to show how flexible the sidecar approach can be, especially when you need to tighten the coupling between processes.
The complete sample is available on GitHub. Training happens in train.py, which builds a simple CNN using TensorFlow and Keras:
The network itself is based on an official tutorial and is compact:
model = models.Sequential() model.add(layers.Conv2D(32, (3, 3), activation="relu", input_shape=(32, 32, 3))) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation="relu")) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation="relu")) model.add(layers.Flatten()) model.add(layers.Dense(64, activation="relu")) model.add(layers.Dense(10))
CIFAR-10 images are 32x32 pixels, each with three color channels (red, green, blue). Pixel values in the dataset are bytes from 0 to 255, which explains the (32, 32, 3) input shape. After training, the script serializes both the model and its weights to a local file.
An Image Server on a Unix Domain Socket
The server loads that serialized model and exposes inference over a Unix domain socket, not HTTP. The wire protocol uses length-prefix encoding:
Each packet begins with a 4-byte length field, followed by a single-byte type, then an arbitrary body. Currently, the server implements two commands:
- 0 - echo: the server returns the same packet back to the client; the body is ignored.
- 1 - classify: the body must be a 32x32 RGB image laid out in row-major order — red channel first (1024 bytes), then green (1024 bytes), then blue (1024 bytes). The server runs inference and responds with the model's predicted label.
A companion Go client can read a PNG from disk, encode it into that format, send it, and print the result. The same client doubles as a benchmark tool:
func runBenchmark(c net.Conn, numIters int) {
// Create a []byte with 3072 bytes.
body := make([]byte, 3072)
for i := range body {
body[i] = byte(i % 256)
}
t1 := time.Now()
for range numIters {
sendPacket(c, messageTypeEcho, body)
cmd, resp := readPacket(c)
if cmd != 0 || len(resp) != len(body) {
log.Fatal("bad response")
}
}
elapsed := time.Since(t1)
fmt.Printf("Num packets: %d, Elapsed time: %s\n", numIters, elapsed)
fmt.Printf("Average time per request: %d ns\n", elapsed.Nanoseconds()/int64(numIters))
}
Measured roundtrip latency over the socket is about 10 microseconds — in line with earlier Unix domain socket benchmarks in Go. For comparison, a single image inference with this model takes roughly 3 ms, so the communication overhead is negligible even if you later move inference to a much faster GPU, where per-image time could drop well below that.
