Go 1.20's PGO in production: a real-world CPU win

Go 1.20 added support for Profile Guided Optimization (PGO) to the compiler, letting it make optimization decisions based on how a program actually behaves in production rather than on static heuristics. At Cloudflare's Observability Team, which runs several Go services on thousands of cores worldwide, even the modest 2–7% savings promised by PGO translate into large numbers of reclaimed CPUs—at effectively no code cost.

The workflow is straightforward: compile a normal binary, deploy it, collect CPU profiles from the running service, then recompile using that profile. The compiler uses the profile data to make targeted improvements—more aggressive inlining of frequently called functions, branch prediction weighted toward common paths, and code layout changes that keep hot code together and reduce cache misses.

Testing PGO on wshim

To validate PGO in a real environment, we applied it to "wshim," a push gateway for telemetry from internal Cloudflare Workers. Wshim runs on every edge server and is invoked each time an internal worker is called, making it CPU-intensive. We track its resource consumption via a dedicated cgroup and Prometheus metrics through cAdvisor.

BLOG-2222 Embedded Image - cOxUzH

Before PGO, wshim consumed over 3,000 cores globally. Even a 2% reduction would return about 60 cores to customer-facing workloads.

Collecting representative profiles

The first challenge was getting good profile samples. With thousands of servers across the world, usage patterns vary by location and time—a data center in peak daytime hours looks very different from one in the middle of the night. We decided to focus on the busiest Tier 1 data centers, where the heaviest load would make wshim's slowest paths most apparent.

To identify the right servers, we queried our Thanos metrics infrastructure for high-CPU instances:

num_profiles="1000"

# Fetch the top n CPU users for wshim across the edge using Thanos.
cloudflared access curl "https://thanos/api/v1/query?query=topk%28${num_profiles}%2Cinstance%3Acontainer_cpu_time_seconds_total%3Arate2m%7Bapp_name%3D%22wshim.service%22%7D%29&dedup=true&partial_response=true" --compressed | jq '.data.result[].metric.instance' -r > "${instances_file}"

Fetching profiles is easy thanks to Go's built-in pprof support over HTTP. Wshim exposes a pprof interface for debugging, which we used to pull profiles from the selected servers via bash:

# For every instance, attempt to pull a CPU profile. Note that due to the transient nature of some data centers
# a certain percentage of these will fail, which is fine, as long as we get enough nodes to form a representative sample.
while read instance; do fetch-pprof $instance –port 8976 –seconds 30' > "${working_dir}/${instance}.pprof" & done < "${instances_file}"

wait $(jobs -p)

We then merged the collected profiles into a single representative file using Go's toolchain:

# Merge the fetched profiles into one.
go tool pprof -proto "${working_dir}/"*.pprof > profile.pprof

This merged profile is committed to the repository alongside other deployment artifacts:

~/cf-repos/wshim ± master
23/01/2024 10:49:08 AEDT❯ tree pgo
pgo
├── README.md
├── fetch-profiles.sh
└── profile.pprof

Finally, we updated the Makefile to pass the -pgo flag during the build:

build:
       go build -pgo ./pgo/profile.pprof -o /tmp/wshim ./cmd/wshim

From there, deploying the PGO-optimized binary is just like any other release.

Measuring the impact

Comparing CPU usage before and after deployment is tricky because wshim's consumption scales with traffic. Load varies by time of day, day of the week, and external events like attacks. Still, we used two approaches to gauge savings.

First, we compared CPU usage immediately before and after the rollout. The release takes just under two hours to reach all Tier 1 data centers, so we used PromQL's offset operator to measure the difference:

BLOG-2222 Embedded Image - yIXlrD
BLOG-2222 Embedded Image - nyWn1Z

This showed roughly 97 fewer cores in use right after the deployment—about a 3.5% reduction. That is in line with the 2–14% range reported in upstream Go PGO documentation.

Second, we compared usage at the same time of day across different weeks. The average over the seven days before the release was 3,067.83 cores; the seven days after showed 2,996.78 cores. That's a savings of about 71 CPUs. Not quite as strong as the immediate comparison, but still substantial—and it proves that PGO can reclaim meaningful computing resources without a single line of code changed.

Next steps

The initial results make a clear case for PGO, but there are several avenues worth exploring. We plan to:

  • Automate profile collection, potentially using continuous profiling.
  • Refine the deployment process to handle the two-step flow (non-PGO build, then PGO build).
  • Improve methods for gathering representative profile samples.
  • Investigate further wins with BOLT or other link-time optimization (LTO) techniques.