Code generation without macros

Go has never had macros or built-in metaprogramming facilities, but that hasn't stopped the language from embracing code generation. Since go generate landed in Go 1.4, it has become a cornerstone of the ecosystem. The Go project itself uses it in dozens of places across the standard library and toolchain.

The design is deliberately minimal. Three pieces do all the work:

  • Generator: any program or script that go generate invokes. A project can have many generators, and one generator can run multiple times.
  • Magic comments: lines in .go files starting at column zero with //go:generate, followed by a command and arguments.
  • go generate: the tool that scans source files for magic comments and executes the specified commands.

That's the entirety of Go's orchestration. Everything else is left to the developer. go generate never runs automatically — not during go build, not when packages are imported, and not for consumers of published modules. Generated code must be checked in and distributed with the source. Running generators is always the developer's explicit job, and it's an expected part of the development workflow before building.

A minimal invocation

To see how the pieces fit together, consider a toy generator called samplegentool. It reads nothing and writes nothing; it only reports how it was invoked:

package main

import (
  "fmt"
  "os"
)

func main() {
  fmt.Printf("Running %s go on %s\n", os.Args[0], os.Getenv("GOFILE"))

  cwd, err := os.Getwd()
  if err != nil {
    panic(err)
  }
  fmt.Printf("  cwd = %s\n", cwd)
  fmt.Printf("  os.Args = %#v\n", os.Args)

  for _, ev := range []string{"GOARCH", "GOOS", "GOFILE", "GOLINE", "GOPACKAGE", "DOLLAR"} {
    fmt.Println("  ", ev, "=", os.Getenv(ev))
  }
}

In a separate module, a file in the mypack package carries a magic comment:

//go:generate samplegentool arg1 "multiword arg"

With the generator's binary on PATH, running go generate ./... from the module root invokes samplegentool for every magic comment found, including in subdirectories:

$ go generate ./...
Running samplegentool go on anotherfile.go
  cwd = /tmp/mymod
  os.Args = []string{"samplegentool", "arg1", "arg2", "arg3", "arg4"}
   GOARCH = amd64
   GOOS = linux
   GOFILE = anotherfile.go
   GOLINE = 1
   GOPACKAGE = mymod
   DOLLAR = $
Running samplegentool go on mymod.go
  cwd = /tmp/mymod
  os.Args = []string{"samplegentool", "arg1", "arg2", "-flag"}
   GOARCH = amd64
   GOOS = linux
   GOFILE = mymod.go
   GOLINE = 3
   GOPACKAGE = mymod
   DOLLAR = $
Running samplegentool go on mypack.go
  cwd = /tmp/mymod/mypack
  os.Args = []string{"samplegentool", "arg1", "multiword arg"}
   GOARCH = amd64
   GOOS = linux
   GOFILE = mypack.go
   GOLINE = 3
   GOPACKAGE = mypack
   DOLLAR = $

The output reveals the essentials. The working directory is always the directory containing the file with the magic comment, so generators always know where they sit in the tree. os.Args shows the full command line, including flags and quoted multi-word arguments. The environment variables passed to the generator — notably GOFILE, the file name relative to the working directory, and GOPACKAGE, the package name — give generators the context they need to produce useful code.

What generators actually do

Generators are unrestricted programs, so they can do anything a program can do. Because generated files are typically checked in, they tend to run rarely and produce files that become ordinary parts of the codebase. The Go standard library has many examples:

  • gob emits repetitive encoding and decoding helpers.
  • math/bits generates fast lookup tables for bitwise operations.
  • Several crypto packages generate hash shuffle patterns and repetitive assembly.
  • Some crypto packages fetch certificates from HTTP URLs during generation — clearly not something to run often.
  • net/http uses generation for HTTP constants.
  • The runtime generates assembly, lookup tables, and mathematical routines.
  • The compiler implementation generates repetitive types and methods for IR nodes.

Notably, at least two places in the standard library — sort and suffixarray — use generators as a pre-generics workaround for near-duplicate code with different types.

Deep dive: stringer

By far the most widely used generator in the Go ecosystem is stringer, which automates the implementation of String() methods for types that should satisfy fmt.Stringer. The most common target is enumerations, like RoundingMode from math/big:

type RoundingMode byte

const (
  ToNearestEven RoundingMode = iota
  ToNearestAway
  ToZero
  AwayFromZero
  ToNegativeInf
  ToPositiveInf
)

A hand-written String() for this type would be a tedious switch statement mapping every value to its textual name. stringer removes that chore. To apply it, add a magic comment alongside the type:

//go:generate stringer -type=RoundingMode

After installing stringer, run go generate on the module:

$ go generate ./...

The command exits quietly and produces a file named roundingmode_string.go:

// Code generated by "stringer -type=RoundingMode"; DO NOT EDIT.

package float

import "strconv"

func _() {
  // An "invalid array index" compiler error signifies that the constant values have changed.
  // Re-run the stringer command to generate them again.
  var x [1]struct{}
  _ = x[ToNearestEven-0]
  _ = x[ToNearestAway-1]
  _ = x[ToZero-2]
  _ = x[AwayFromZero-3]
  _ = x[ToNegativeInf-4]
  _ = x[ToPositiveInf-5]
}

const _RoundingMode_name = "ToNearestEvenToNearestAwayToZeroAwayFromZeroToNegativeInfToPositiveInf"

var _RoundingMode_index = [...]uint8{0, 13, 26, 32, 44, 57, 70}

func (i RoundingMode) String() string {
  if i >= RoundingMode(len(_RoundingMode_index)-1) {
    return "RoundingMode(" + strconv.FormatInt(int64(i), 10) + ")"
  }
  return _RoundingMode_name[_RoundingMode_index[i]:_RoundingMode_index[i+1]]
}

This file embodies one of stringer's several code generation strategies. For a single consecutive run of enumeration values, it packs all names into one long string constant (_RoundingMode_name) and uses an index array (_RoundingMode_index) to slice out each name. Value 2 (ToZero) indexes to position 26 in the name string, with the end determined by the next index value at 32.

The generated String() method also includes a fallback for values added after generation: it prints RoundingMode(N) rather than a missing name. This matters because nothing in the toolchain enforces that generated code stays in sync with source changes — that responsibility belongs to the developer.

The odd-looking func _() is a compile-time guard. It has no runtime effect; it exists solely to catch a dangerous mistake. If an existing enumeration value is changed without rerunning go generate, the generated String() could silently return wrong names. The guard forces an out-of-bounds array compilation error instead, making the problem visible immediately.

stringer accepts a few flags worth understanding:

$ stringer -help
Usage of stringer:
  stringer [flags] -type T [directory]
  stringer [flags] -type T files... # Must be a single package
For more information, see:
  https://pkg.go.dev/golang.org/x/tools/cmd/stringer
Flags:
  -linecomment
      use line comment text as printed text when present
  -output string
      output file name; default srcdir/<type>_string.go
  -tags string
      comma-separated list of build tags to apply
  -trimprefix prefix
      trim the prefix from the generated constant names
  -type string
      comma-separated list of type names; must be set

The -type flag selects which types in the package get a String() method. The default -output names the generated file after the type, yielding roundingmode_string.go.

One subtlety: when invoked from a magic comment, stringer does not read GOFILE and does not analyze the file containing the comment. Instead, the tool uses golang.org/x/tools/go/packages to load the entire package from the current working directory. This makes sense because constants can live in different files than the type declaration. In Go, the package — not the file — is the unit of interest.

Keeping generators inside the module

So far we've assumed that generators are found in PATH when go generate runs. But a common scenario is a module that carries its own generator, useful only to developers working on that specific module. You want someone to clone the code, run go generate, and build without first installing any extra tools.

Go handles this cleanly via go run, which is perfect for running generator programs that live as .go files somewhere in the module tree. A magic comment can invoke the generator directly from its source location:

package mypack

//go:generate go run gen.go arg1 arg2

func PackFunc() string {
  return "insourcegenerator/mypack.PackFunc"
}

The generator itself is just a small Go program in package main. The one notable detail is the build constraint that tells the Go toolchain to exclude this file from the package build:

//go:build ignore

package main

import (
  "fmt"
  "os"
)

func main() {
  // ... same main() as the simple example at the top of the post
}

The gen.go file is not part of the package — it's standalone code meant to be executed by go generate, not compiled into the module. The standard library contains many such in-tree generator programs that follow this pattern.

A typical layout involves three files coexisting in the same directory:

  • The source file holds package code plus a magic comment invoking a generator via go run.
  • The generator is a single package main .go file, usually carrying a //go:build ignore constraint so it isn't compiled into the package. It's launched by the magic comment and produces the generated file.
  • The generated file is emitted by the generator. Often its name derives from the source file (e.g. pack.go → pack_gen.go) or uses a gen prefix. Its code belongs to the same package as the source file, and it may define unexported symbols that the source file references directly.

None of this is enforced by tooling — it's just a common convention. Projects can structure generation differently, such as a single generator producing code for multiple packages.

Less common features

Aliasing commands with -command

The -command flag defines aliases for go:generate lines—useful to shorten a multi-word generator command that appears several times. Its original purpose was likely to reduce go tool yacc to just yacc:

//go:generate -command yacc go tool yacc

After that, yacc can be invoked with a single word. Notably, go tool yacc was removed from the core toolchain in Go 1.8, and the flag sees no real use in the main Go repository or x/tools outside of tests for go generate itself.

Filtering with -run

The -run flag applies to the go generate command itself, letting you select a subset of generators to execute. If a project has three samplegentool invocations, you can pick just one:

$ go generate -run multi ./...
Running samplegentool go on mypack.go
  cwd = /tmp/mymod/mypack
  os.Args = []string{"samplegentool", "arg1", "multiword arg"}
   GOARCH = amd64
   GOOS = linux
   GOFILE = mypack.go
   GOLINE = 3
   GOPACKAGE = mypack
   DOLLAR = $

This is handy for debugging: in a large project with multiple generators, you can run only the relevant one and keep the edit-run cycle fast.

The DOLLAR environment variable

Among the environment variables automatically passed to generators, DOLLAR stands out—it doesn't seem to have an obvious purpose, and it has no use in the Go source tree. Its origin goes back to a commit by Rob Pike, where the goal was passing a literal $ character into a generator without fighting shell escaping rules. This matters when go generate invokes a shell script that expects a regexp argument.

The effect is visible with the samplegentool generator. A magic comment containing $somevar:

//go:generate samplegentool arg1 $somevar

causes the generator to see empty arguments:

os.Args = []string{"samplegentool", "arg1", ""}

Because the shell interprets $somevar as a variable reference to something that doesn't exist. Using DOLLAR instead:

//go:generate samplegentool arg1 ${DOLLAR}somevar

yields the intended literal text:

os.Args = []string{"samplegentool", "arg1", "$somevar"}