Writing multi-package analysis tools for Go - Eli Bendersky's website

From Idea to Tool: Analyzing Real-World Go Code

When I needed to catalog the different kinds of type embedding used across Go's standard library, I faced a choice: rely on my admittedly incomplete mental model of the codebase, or write a small tool to do the searching for me. The latter approach—building a code analysis tool—turned out to be surprisingly straightforward thanks to Go's existing tooling infrastructure.

The core challenge is that parsing alone is insufficient for this task. Consider this struct from my earlier writing on embedding:

type StatsConn struct {
  net.Conn

  BytesRead uint64
}

From the syntax alone, we can see that net.Conn is an embedded field. But determining whether net.Conn is an interface or a struct requires type information—and in the general case, that type lives in a different package or even a different module. A practical analysis tool must therefore handle cross-package, cross-module type checking. This is where the x/tools/go/packages package comes in.

The One-Stop Shop for Package Loading

The x/tools/go/packages package (which I'll call XTGP) is designed to take on the heavy lifting of package analysis. It handles parsing, type checking, and—optionally—loading and type-checking a package's dependencies. As the newest iteration (2018) in a line of similar packages, XTGP is now the standard foundation for multi-package analysis; it also underlies the x/tools/go/analysis framework used by tools like go vet.

Configuration happens through packages.Load with a packages.Config object:

import "golang.org/x/tools/go/packages"

const mode packages.LoadMode = packages.NeedName |
  packages.NeedTypes |
  packages.NeedSyntax |
  packages.NeedTypesInfo

func main() {
  flag.Usage = func() {
    out := flag.CommandLine.Output()
    fmt.Fprintln(out, "usage: find-embeddings [options] <module dir>\n")
    fmt.Fprintln(out, "Options:")
    flag.PrintDefaults()
  }

  pattern := flag.String("pattern", "./...", "Go package pattern")
  flag.Parse()
  if flag.NArg() != 1 {
    log.Fatal("Expecting a single argument: directory of module")
  }

  var fset = token.NewFileSet()
  cfg := &packages.Config{Fset: fset, Mode: mode, Dir: flag.Args()[0]}
  pkgs, err := packages.Load(cfg, *pattern)
  if err != nil {
    log.Fatal(err)
  }

  for _, pkg := range pkgs {
    findInPackage(pkg, fset)
  }
}

The Mode field is the critical part of this configuration. It's tempting to set it to load everything, but that can be slow for large projects. We don't need NeedTypesInfo or NeedDeps, which would force the type-checking of all transitive dependencies. Since we only need type information about dependencies' exported types—information Go makes available cheaply to support its fast parallel builds—we can keep the load mode lean.

Once the packages load, we get a slice of packages.Package values. For each one, we run our core analysis function, findInPackage:

func findInPackage(pkg *packages.Package, fset *token.FileSet) {
  for _, fileAst := range pkg.Syntax {
    ast.Inspect(fileAst, func(n ast.Node) bool {
      if structTy, ok := n.(*ast.StructType); ok {
        findInFields(structTy.Fields, n, pkg.TypesInfo, fset)
      } else if interfaceTy, ok := n.(*ast.InterfaceType); ok {
        findInFields(interfaceTy.Methods, n, pkg.TypesInfo, fset)
      }

      return true
    })
  }
}

This function has two distinct jobs. First, it uses ast.Inspect to walk every AST node, looking specifically at *ast.StructType and *ast.InterfaceType nodes. Second, it handles the structural difference between these types: structs expose their fields via the Fields field, whereas interfaces use Methods.

From there, the real logic lives in findInFields:

func findInFields(fl *ast.FieldList, n ast.Node, tinfo *types.Info, fset *token.FileSet) {
  type FieldReport struct {
    Name string
    Kind string
    Type types.Type
  }
  var reps []FieldReport

  for _, field := range fl.List {
    if field.Names == nil {
      tv, ok := tinfo.Types[field.Type]
      if !ok {
        log.Fatal("not found", field.Type)
      }

      embName := fmt.Sprintf("%v", field.Type)

      _, hostIsStruct := n.(*ast.StructType)
      var kind string

      switch typ := tv.Type.Underlying().(type) {
      case *types.Struct:
        if hostIsStruct {
          kind = "struct (s@s)"
        } else {
          kind = "struct (s@i)"
        }
        reps = append(reps, FieldReport{embName, kind, typ})
      case *types.Interface:
        if hostIsStruct {
          kind = "interface (i@s)"
        } else {
          kind = "interface (i@i)"
        }
        reps = append(reps, FieldReport{embName, kind, typ})
      default:
      }
    }
  }

  if len(reps) > 0 {
    fmt.Printf("Found at %v\n%v\n", fset.Position(n.Pos()), nodeString(n, fset))

    for _, report := range reps {
      fmt.Printf("--> field '%s' is embedded %s: %s\n", report.Name, report.Kind, report.Type)
    }
    fmt.Println("")
  }
}

This function iterates over a field list, looking for unnamed (embedded) fields. For each one, it inspects the underlying type declaration to determine whether it's a struct or an interface. This classification step is where the cross-package type analysis earns its keep—without it, we'd have no idea whether an embedded type from another module is a struct or an interface.

An Alternative: The go/analysis Framework

XTGP gives you direct control, but the go/analysis framework offers a more streamlined approach, eliminating some boilerplate:

import "golang.org/x/tools/go/analysis"
import "golang.org/x/tools/go/analysis/singlechecker"

var EmbedAnalysis = &analysis.Analyzer{
  Name: "embedanalysis",
  Doc:  "reports embeddings",
  Run:  run,
}

func main() {
  singlechecker.Main(EmbedAnalysis)
}

func run(pass *analysis.Pass) (interface{}, error) {
  for _, file := range pass.Files {
    ast.Inspect(file, func(n ast.Node) bool {
      if structTy, ok := n.(*ast.StructType); ok {
        findInFields(structTy.Fields, n, pass.TypesInfo, pass.Fset)
      } else if interfaceTy, ok := n.(*ast.InterfaceType); ok {
        findInFields(interfaceTy.Methods, n, pass.TypesInfo, pass.Fset)
      }

      return true
    })
  }

  return nil, nil
}

Notice how much shorter the main function becomes. By delegating to go/analysis and its singlechecker helper, we avoid explicitly initializing go/packages and handling command-line flags ourselves.

The analysis logic is nearly identical to the XTGP version. The run function plays the role of findInPackage, iterating over pass.Files (instead of pkg.Syntax) and invoking the same shared findInFields for each struct or interface declaration.

Choosing an Approach

Both methods achieve the same goal, but each has its strengths. Using XTGP directly offers more control over package loading configuration and tool CLI behavior, and keeps the process more transparent—fewer black boxes. The go/analysis route involves slightly less code and integrates well with an ecosystem of existing analysis passes; you can also give it code that runs many analyses and shares information between them.

Either way, Go's tooling ecosystem handles the tedious part of figuring out how a project is assembled from modules and packages, giving you a fully type-checked AST to work with. For more specialized analyses that require an intermediate representation—for instance, an SSA form—x/tools/go/ssa can build it on top of these type-checked packages.