Why Go benchmarks mislead
Go’s testing package gives developers a strong benchmarking foundation, but benchmarking itself is deceptively hard. The pitfalls below are common across languages and environments, not just Go — though Go’s tooling makes them easy to hit if you’re not careful. We assume basic fluency with writing Go benchmarks; the package docs cover the fundamentals.
Measuring the wrong operation
Consider a benchmark for slices.Sort, available from Go 1.21:
const N = 100_000
func BenchmarkSortIntsWrong(b *testing.B) {
ints := makeRandomInts(N)
b.ResetTimer()
for i := 0; i < b.N; i++ {
slices.Sort(ints)
}
}
func makeRandomInts(n int) []int {
ints := make([]int, n)
for i := 0; i < n; i++ {
ints[i] = rand.Intn(n)
}
return ints
}
This looks reasonable, but slices.Sort sorts in place. After the first run, the slice is already ordered, so every later iteration sorts a sorted slice. The fix is to build a fresh random slice per iteration:
func BenchmarkSortInts(b *testing.B) {
for i := 0; i < b.N; i++ {
b.StopTimer()
ints := makeRandomInts(N)
b.StartTimer()
slices.Sort(ints)
}
}
That corrected version runs almost 100× slower on a typical machine — because it actually does real work each time.
Timer hygiene matters
Both benchmarks above deliberately stop and start the benchmark timer around setup work. The harness measures the total execution time of the whole Benchmark* function across many runs, then divides by b.N. If you drop the b.StopTimer()/b.StartTimer() pair from BenchmarkSortInts, the per-op numbers jump up — slice creation now costs b.N times too.
That error doesn’t always cancel out when comparing two implementations. Because of Amdahl’s law, setup overhead skews speed-up ratios even when the same flawed pattern applies to both sides.
Compiler optimizations that silently invalidate results
Update (2025-02-24): Go 1.24’s new B.Loop API is designed to avoid these problems — you should no longer write explicit for loops with b.N.
Compilers don’t treat benchmark functions specially. Go will optimize them like any other code, and a benchmark can report confident but meaningless numbers. Take this example targeting isCond:
func isCond(b byte) bool {
if b%3 == 1 && b%7 == 2 && b%17 == 11 && b%31 == 9 {
return true
}
return false
}
func BenchmarkIsCondWrong(b *testing.B) {
for i := 0; i < b.N; i++ {
isCond(201)
}
}
Running it yields:
BenchmarkIsCondWrong-8 1000000000 0.2401 ns/op
Two errors hide here. First, the input is a compile-time constant; isCond is simple enough to inline, so the compiler can constant-fold the body and replace the call with its result. Second, even with a variable input, the call’s return value is never used, so the compiler may delete the work entirely. The disassembly confirms both:
JMP BenchmarkIsCondWrong_pc7
BenchmarkIsCondWrong_pc4:
INCQ CX
BenchmarkIsCondWrong_pc7:
CMPQ 416(AX), CX
JGT BenchmarkIsCondWrong_pc4
RET
The loop is empty — CX is the loop variable and 416(AX) reads b.N. That explains the 0.24 ns/op.
A more subtle version of the same trap:
func countCond(b []byte) int {
result := 0
for i := 0; i < len(b); i++ {
if isCond(b[i]) {
result++
}
}
return result
}
func BenchmarkCountWrong(b *testing.B) {
inp := getInputContents()
b.ResetTimer()
for i := 0; i < b.N; i++ {
countCond(inp)
}
}
func getInputContents() []byte {
n := 400000
buf := make([]byte, n)
for i := 0; i < n; i++ {
buf[i] = byte(n % 32)
}
return buf
}
Here isCond’s output feeds into countCond, but countCond’s result is discarded. The compiler can inline both functions, prove the loop body has no side effects and no externally visible result, and hollow out the entire inner loop — leaving just the loop skeleton. The benchmark still shows a runtime that grows with input size, so input-scaling smoke tests won’t catch the problem.
Keeping the compiler honest
Two techniques block unwanted optimization. The first is runtime.KeepAlive, which claims a value is still needed even when the compiler can prove otherwise:
func BenchmarkCountKeepAlive(b *testing.B) {
inp := getInputContents()
b.ResetTimer()
result := 0
for i := 0; i < b.N; i++ {
result += countCond(inp)
}
runtime.KeepAlive(result)
}
Results now show real work:
BenchmarkCountWrong-8 12481 95911 ns/op BenchmarkCountKeepAlive-8 4143 285527 ns/op
The second, riskier route uses a package-level exported sink variable:
var Sink int
func BenchmarkCountSink(b *testing.B) {
inp := getInputContents()
b.ResetTimer()
for i := 0; i < b.N; i++ {
Sink += countCond(inp)
}
}
The compiler rarely removes writes to exported globals because any package could reference them; but cross-package analysis could still prove it redundant. When results look suspicious, check the assembly rather than trusting the numbers.
Standard-library progress
The Go team has two active proposals on this front: issue 61179, for a targeted low-overhead “keep this value” helper in testing; and the broader API redesign sketched in issue 61515.
Abusing b.N
Two common misuses of b.N produce bizarre outputs. First, omitting the loop entirely:
import (
"crypto/rand"
"testing"
)
func BenchmarkRandPrimeWrongNoLoop(b *testing.B) {
rand.Prime(rand.Reader, 200)
}
crypto/rand.Prime at 200 bits should take roughly a millisecond per call; this benchmark reports fractions of a nanosecond.
The second misuse is more insidious:
func BenchmarkRandPrimeWrongUseI(b *testing.B) {
for i := 0; i < b.N; i++ {
rand.Prime(rand.Reader, i)
}
}
Here the loop body uses i (which depends on b.N) as the prime size. On many machines this benchmark never finishes. Understanding why requires a look at the harness: it starts with a small b.N, measures, then scales up — at most 100× per step, capped at 1 billion — until the target -benchtime duration is reached.
Both failure modes follow:
- No loop: each invocation does one unit of work regardless of
b.N. The harness keeps growingb.Ntoward its upper bound while execution time stays constant; dividing that time by the hugeb.Nyields meaningless sub-nanosecond figures. - Input bound to
b.N: smallb.Nvalues run quickly, letting the harness raise the repetition count — but largerb.Nalso makesrand.Primefar slower. The combined effect is roughly quadratic growth; the benchmark may eventually finish, but after minutes or hours.
The design in issue 61515 targets exactly this class of error. The pending range-over-int proposal would also help, allowing for range b.N without an explicit index variable.
