Google's newest family of multimodal LLMs is reachable through an API with a generous free tier, and Google ships SDKs for several popular languages, Go among them. Getting a Go program to ask the model questions about images mixed with text takes very little code.

Two turtles, one question

The example task is to have the model explain how two images of turtles differ.

first turtle image second turtle image

Google AI SDK setup

Access to the model requires only an API key, the same pattern OpenAI's API uses. The Go SDK is hosted at https://github.com/google/generative-ai-go, documented at https://pkg.go.dev/github.com/google/generative-ai-go, and comes with a useful set of examples.

package main

import (
  "context"
  "encoding/json"
  "fmt"
  "log"
  "os"

  "github.com/google/generative-ai-go/genai"
  "google.golang.org/api/option"
)

func main() {
  ctx := context.Background()
  client, err := genai.NewClient(ctx, option.WithAPIKey(os.Getenv("API_KEY")))
  if err != nil {
    log.Fatal(err)
  }
  defer client.Close()

  model := client.GenerativeModel("gemini-pro-vision")

  imgData1, err := os.ReadFile("../images/turtle1.png")
  if err != nil {
    log.Fatal(err)
  }

  imgData2, err := os.ReadFile("../images/turtle2.png")
  if err != nil {
    log.Fatal(err)
  }

  prompt := []genai.Part{
    genai.ImageData("png", imgData1),
    genai.ImageData("png", imgData2),
    genai.Text("Describe the difference between these two pictures, with scientific detail"),
  }
  resp, err := model.GenerateContent(ctx, prompt...)

  if err != nil {
    log.Fatal(err)
  }

  bs, _ := json.MarshalIndent(resp, "", "    ")
  fmt.Println(string(bs))
}

Because the API is multimodal, the SDK supplies helper types such as genai.ImageData and genai.Text, which wrap inputs type-safely. Running the sample dumps the model's reply as a JSON object; the part that matters is the explanation itself.

"Content": {
  "Parts": [
    "The first picture is of a tortoise, which is a reptile characterized by
    its hard shell. The second picture is of a sea turtle, which is a reptile
    characterized by its flippers and streamlined shell. Tortoises are
    terrestrial animals, while sea turtles are marine animals. Tortoises have
    a domed shell, while sea turtles have a flattened shell. Tortoises have
    thick, scaly skin, while sea turtles have smooth, leathery skin. Tortoises
    have short legs with claws, while sea turtles have long flippers.
    Tortoises have a slow metabolism and can live for over 100 years, while
    sea turtles have a faster metabolism and typically live for around 50
    years."
  ],
  "Role": "model"
},

Switching to the Vertex SDK

GCP customers whose project already has billing and the rest of the setup in place may prefer the Vertex Go SDK. Code written against the Google AI SDK transfers almost untouched: only the import line changes, from

"github.com/google/generative-ai-go/genai"

to

"cloud.google.com/go/vertexai/genai"

and client construction differs because authentication differs. Under Vertex the client is created like this:

client, err := genai.NewClient(ctx, os.Getenv("GCP_PROJECT_ID"), "us-central1")

Here GCP_PROJECT_ID is an environment variable holding your GCP project, and the location/region is set according to your preference. Everything else in the program stays the same.

Two SDKs exist because the two products don't always offer identical features; the GCP one, for instance, may let you read data straight from your storage buckets or database tables.

Samples and follow-up

All the code in the post, along with the sample images, lives in the corresponding code-for-blog repository on GitHub. A later post covers reaching the Gemini models through langchaingo.