Gemma, Ollama and LangChainGo - Eli Bendersky's website

Gemma Meets Ollama and LangChainGo

Google's open Gemma model landed yesterday, and local execution is surprisingly painless. Ollama already supports it, so pulling and running the model is a single command:

$ ollama run gemma

That command downloads the model, and Gemma then appears as a standard local REST endpoint. The setup mirrors what I documented before for Ollama in Go, so existing code paths work almost unchanged. LangChainGo's Ollama provider handles Gemma transparently, and I've updated my prior samples with a --model gemma flag. A minimal interaction looks like this:

package main

import (
  "context"
  "flag"
  "fmt"
  "log"

  "github.com/tmc/langchaingo/llms"
  "github.com/tmc/langchaingo/llms/ollama"
)

func main() {
  modelName := flag.String("model", "", "ollama model name")
  flag.Parse()

  llm, err := ollama.New(ollama.WithModel(*modelName))
  if err != nil {
    log.Fatal(err)
  }

  query := flag.Args()[0]
  ctx := context.Background()
  completion, err := llms.GenerateFromSinglePrompt(ctx, llm, query)
  if err != nil {
    log.Fatal(err)
  }

  fmt.Println("Response:\n", completion)
}

Run it with:

$ go run ollama-completion-arg.go --model gemma "what should be added to 91 to make -20?"
Response:
 The answer is -111.

91 + (-111) = -20

Performance is respectable for CPU inference. The default 7B Gemma model not only benchmarks better than the default 7B llama2, it also runs roughly 30% faster on my hardware.

Skipping the Framework

LangChainGo isn't mandatory here. Ollama ships its own Go API, usable externally without any wrapper. An equivalent program can talk to Ollama directly:

package main

import (
  "context"
  "flag"
  "fmt"
  "log"

  "github.com/jmorganca/ollama/api"
)

func main() {
  modelName := flag.String("model", "", "ollama model name")
  flag.Parse()

  client, err := api.ClientFromEnvironment()
  if err != nil {
    log.Fatal(err)
  }

  req := &api.GenerateRequest{
    Model:  *modelName,
    Prompt: flag.Args()[0],
    Stream: new(bool), // disable streaming
  }

  ctx := context.Background()
  var response string
  respFunc := func(resp api.GenerateResponse) error {
    response = resp.Response
    return nil
  }

  err = client.Generate(ctx, req, respFunc)
  if err != nil {
    log.Fatal(err)
  }

  fmt.Println("Response:\n", response)
}

This all works because each layer sticks to a defined contract: Gemma packages into Ollama's model format, Ollama exposes a stable REST interface, and LangChainGo abstracts that interface for Go code.