Finding and Fixing Performance Regressions in Ruby

Large Rails applications inevitably accumulate inefficient code. At Shopify's scale, with thousands of daily commits to a monolithic codebase, performance regressions can become noticeable long before the offending commit is identified. The challenge is two-fold: effectively pinpointing the cause of slowness, and proving a proposed fix actually helps.

Profiling gives you observability into runtime behavior, while benchmarking validates whether a change improves performance. Both are essential for maintaining a healthy codebase.

Profiling: Finding the Bottleneck

Profiling collects runtime metrics—such as method call frequency and duration—to help you locate performance bottlenecks. A profiler's output can be rendered as flat profiles, call graphs, or flamegraphs. Static analysis and code reviews only go so far; profiling reveals what the program actually does during execution.

Choosing a Profiling Approach

Decide what you need to measure first. Do you want elapsed time for a code block, or object allocations within it? Do you want per-method granularity, or is an aggregate sufficient? Note also that elapsed time breaks down into CPU time versus wall time.

For elapsed time, you can measure start and end points around a block. For per-method granularity, Ruby's TracePoint API hooks into every method call. For object allocation tracing, the ObjectSpace module can trace allocations or dump the heap. But rather than building custom solutions, consider one of these established profilers:

  • rbspy — Samples stack frames from a running Ruby process without requiring any code instrumentation. Profile an existing process by PID:

rbspy record —pid $PID

  • stackprof — Samples stack frames from an instrumented code block, ideal for profiling a specific portion of your code:

profile = StackProf.run(mode: :cpu) do
  # Code to profile
end

  • rack-mini-profiler — A full-featured gem for Rack apps that combines call-stack sampling with memory profiling (GC statistics, allocation counts). Uses stackprof and memory_profiler internally.
  • app_profiler — A lightweight alternative to rack-mini-profiler from Shopify. Provides Rack middleware for request-level profiling plus block-level profiling. Profiles can be stored in configurable backends (e.g., Google Cloud Storage) and visualized with Speedscope flamegraphs.

Shopify built app_profiler because rack-mini-profiler offered more features than needed. Under the hood it also uses stackprof, and now powers on-demand remote profiling of production requests.

Case Study: GC Cycles in Production

Profiling in production once exposed a serious issue with a cart item having a very large quantity. Requests were consuming excessive CPU time due to Ruby allocating so many objects that garbage collection ran repeatedly.

A flamegraph of a similar slow request showed roughly 500ms of CPU time, with GC operations highlighted in chunks interleaved with normal operations. GC alone consumed about 35% of that CPU time—strong evidence of excessive object allocation. Without profiling, identifying this type of problem would have been far more difficult.

Benchmarking: Proving Your Fix

Once a bottleneck is found, benchmarking demonstrates which code path performs better. The simplest form measures wall time for a block using Ruby's standard Benchmark.realtime. But the standard library offers more:

  • bm — Displays detailed timing breakdowns: user CPU time, system CPU time, total CPU time, and real (wall) time. The user/system split distinguishes work in user space from kernel space.
  • bmbm — Identical to bm but adds a rehearsal run. This warmup step primes caches and similar mechanisms for more stable, reproducible measurements.

The widely used benchmark-ips gem takes a different approach. Instead of timing fixed iterations, it measures iterations per second. A simple script requires inline bundler setup since the gem isn't in the standard library:

require 'benchmark/ips'
Benchmark.ips do |x|
  x.report("method") { method }
end

Output shows warmup iterations per 100 milliseconds and the number of times the block executed within the measurement period (default 5 seconds).

Comparing Code Paths

The real value of benchmarking appears when comparing alternatives. Add multiple report blocks and call compare! to print which implementation is slower and by how much:

Benchmark.ips do |x|
  x.report("method_a") { method_a }
  x.report("method_b") { method_b }
  x.compare!
end

This instantly quantifies performance differences, backing up proposals with evidence rather than intuition. Running the same benchmark against pre- and post-patch versions of code measures improvement at a particular code path over time, catching regressions early.

Developers often propose optimizations without proof. Benchmarking provides that proof—whether comparing two solutions for a new fix or verifying that a recent change didn't degrade performance.

Benchmarking Inside a Rails App

Trivial benchmark examples are easy to follow but rarely reflect the complexity of real application code. In a framework like Ruby on Rails, loading the entire framework and application context into a standalone script can be a hurdle. Rails 6.1 addresses this with a built-in benchmark generator.

Running bin/rails generate benchmark my_benchmark creates a file at script/benchmarks/my_benchmark.rb. Because the generated script loads via the Rails app’s Gemfile, an inline gemfile is unnecessary.

A concrete example helps illustrate the process. Consider a benchmark that subclasses Order and caches the calculation that totals all line item prices. Whether such a cache is worthwhile isn’t obvious without measurement — the base implementation could already be reasonably fast.

The benchmark script defines an Order subclass with a cached total, then compares it against the original implementation on an order with four line items. Running the script shows roughly a 50x improvement for that simple case. Orders with more line items see even larger gains.

An unabridged version of the script provides full context for how the benchmark is structured and executed.

One caveat to keep in mind while benchmarking is micro-optimization. These are tweaks so small that the maintenance cost of the code change outweighs the performance gain. They can be acceptable on hot code paths, but broader performance issues should be tackled first.

What Rails Pull Requests Look Like

Ruby on Rails, like most open source projects, expects performance-related pull requests to include benchmarks. This is particularly true for changes in performance-sensitive areas such as Active Record's query building or Active Support's cache stores. Across the Rails project, benchmark-ips is the standard tool for such comparisons.

A few merged pull requests show the pattern in practice:

  • PR #36052 changes how primary keys are accessed on Active Record instances, refactoring class method calls into instance variable references. It includes before and after benchmark results with an explanation of why the refactor is needed.
  • PR #38401 modifies model attribute assignment so that key stringification of attribute hashes is no longer required. The benchmark script covers multiple scenarios, which matters because record creation and updating sit at the core of most Rails applications.
  • PR #34197 reduces object allocations in ActiveRecord#respond_to?. Its memory benchmark compares total allocations before and after the patch, showing a calculated diff. Fewer allocations means less time spent assigning objects to memory blocks and better overall performance.

Benchmarking as a Habit

Slow code shows up in every codebase eventually. When it does, the focus should be on the fix, not on who introduced the regression. Profiling and benchmarking are the tools for finding and resolving those performance problems.

Ruby itself prioritizes developer experience over server efficiency, so idiomatic, maintainable Ruby is often not the fastest option. Shopify has written plenty of slow code, frequently for good reasons. The practical takeaway: profile and benchmark responsibly, watch out for micro-optimizations, and address the largest performance issues first.

Further Reading