Why Our Monolith Was Stalling on GC

At the start of this year, we focused on reducing the latency impact of Ruby's garbage collector (GC) in Shopify's monolith. The work spanned several rounds of improving logging and metrics, forming hypotheses, then shipping and evaluating changes—sometimes tweaking or reverting them. While this narrative may read as a straight line from problem to resolution, the actual process included dead ends and unexpected results, as expected when optimizing a dynamic system.

Every Ruby object is allocated with an age starting at 0. Each time the GC runs, surviving objects have their age incremented. At a maximum age of 3, an object is promoted to the old generation. The GC then decides, based on multiple heuristics, whether to run a major or minor mark. A major mark traverses all objects, while a minor mark only sweeps the young subset, making it quick. In our case, minor marks averaged around 3ms, while major marks could take over 4 seconds.

Because Ruby's GC is also incremental, major marks don't have to run in one continuous block. The collector can mark part of the heap, pause, allow Ruby code to execute, and then resume later. Still, when the GC decides to work, it typically does so during a request, pausing all other execution. These pauses add directly to request latency, making merchants and buyers wait longer.

Diagnosing the Tail Latency

When we first examined the problem, the latency distribution for time spent in GC per request looked alarming, as illustrated below:

These graphs represent the distribution of GC time for requests with GC logs enabled, which is 1% of all traffic. The median request spent just 3ms in GC—fine. The p99 request tallied around 323ms, which is already poor. But in the tail, some requests spent multiple seconds doing GC work. That tail was our primary target.

The distribution owed much to the generational nature of the collector. Minor marks are cheap, but major marks, which inspect the entire heap, are slow. Our main goal became reducing the frequency of major marks. This required identifying what triggered them in the first place.

Existing GC Tuning Was Stale

The monolith already had some GC environment variables set from a few years ago. These were documented in the Ruby man page, but clearly needed updating. We suspected they were outdated or not suited to our current workload. However, we didn't rely on intuition alone—each change was measured carefully.

A Caution on Tuning

You should never blindly apply GC settings without first establishing a solid way to measure impact. Changing GC configuration can radically affect application performance, either improving it or making it much worse. Ruby's default GC settings are decent choices for the vast majority of applications—we strongly recommend leaving them alone unless you have strong evidence that GC is materially harming performance.

Choosing Our Battles: The oldmalloc Limit

To find out what was forcing major marks, we logged GC.latest_gc_info(:major_by). That showed around 75% of major marks came from :oldmalloc and almost 25% from :shady.

oldmalloc tracks bytes allocated through malloc. When the counter passes a threshold, a major GC fires and resets the counter to zero. The theory is that frequent allocation by old objects signals reference churn worth cleaning up. The current state is visible via GC.stat(:oldmalloc_increase_bytes) and the trigger point via GC.stat(:oldmalloc_increase_bytes_limit).

Those limits are also configurable with RUBY_GC_OLDMALLOC_LIMIT, RUBY_GC_OLDMALLOC_LIMIT_MAX, and RUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR. Our workload didn't fit this heuristic, so we disabled it by setting both the LIMIT and LIMIT_MAX to 128GB. There's a catch: without setting LIMIT_MAX, the limit actually grows unbounded by GROWTH_FACTOR after each major GC, which means our old config had no real effect at all.

Rolling this out to half of our production processes cut major marking by roughly 20%:

The reason the reduction wasn't the full 75% is that a major GC resets many heuristics at once. Once one cause is removed, another limit simply starts being hit more often. After this change, nofree became the dominant reason for major marks.

Slot Preallocation and nofree

The nofree trigger comes from the heap being out of empty object slots. On allocation failure, Ruby runs a minor mark to free slots; if that isn't enough, it escalates to a major mark and finally grows the heap by adding new slots. Environment variables control these thresholds, letting us shape when this rescue path fires.

In a steady-state process, nofree disappears once enough slots exist to recycle. But Shopify deploys nearly every 30 minutes, so most processes spend their lives in the warm-up phase where this happens repeatedly. A natural fix for that is RUBY_GC_HEAP_INIT_SLOTS, which preallocates slots at boot. Previously we used 1.6M slots; production instrumentation through GC.start(:heap_live_slots) indicated we really needed about 15M.

On Ruby 3.1 this would have been a simple setting. But we moved to Ruby 3.2 in January, which brought variable width allocation. The heap now has five slot size pools, inspectable through GC.stat_heap. For each pool, heap_eden_slots shows the total usable slots, while force_major_gc_count tracks how often that specific pool forced a major mark. With that data we could compute a sensible preallocation per pool.

Unfortunately, setting RUBY_GC_HEAP_INIT_SLOTS in Ruby 3.2 splits the initial slots proportionally to size pool capacity, not production reality. A warm worker's live object counts per pool look very different from that split. Uniform preallocation would either waste memory on unused pools or under-allocate others. Neither option was acceptable.

Working Around the Pool Split

Rather than accept the proportional split, we simulated fine-grained preallocation by loading a temporary workaround at boot in config/boot.rb. This let us drive slot counts per size pool through environment variables. Deploying it to production had an immediate positive effect on GC tail latency (gc_tuned_2):

After tuning the numbers a few more times, we had a proper Ruby feature merged upstream and backported it to our Ruby 3.2 builds. This helped, but overall time in GC was still higher than we liked.

Shady Objects and Missing Write Barriers

The remaining significant trigger was shady. Its exact meaning isn't intuitive without a look at how generational GC and write barriers interact. To escape to the old generation and avoid being marked on every minor GC, an object has to promise to notify the GC whenever it changes a reference to another object. That notification is a write barrier. Getting them wrong causes subtle bugs when slots get reused, so several C extensions and core types still avoid them.

When an old object references a young one, the young one is normally promoted to the old generation immediately. That's impossible for an object without a write barrier, so the GC instead tracks it in a list to make sure it gets marked on each minor. Those tracked objects are shady, known internally these days as uncollectible_wb_unprotected_object. The keyword in GC.latest_gc_info remains shady. Too many accumulated shady objects automatically trigger a major mark to clean some up. The internals are exposed through GC.stat(:remembered_wb_unprotected_objects) and GC.stat(:remembered_wb_unprotected_objects_limit), but unlike oldmalloc, there is no proper environment variable to adjust that limit.

Raising the limit would just convert slower minor marks into less frequent major marks—a losing trade. So instead we fixed the root cause: missing write barriers. Dumps from ObjectSpace.dump_all in production helped identify frequent offenders across Ruby and popular gems. We contributed close to forty upstream fixes, including notable ones for Time, BigDecimal, Method, oj, google-protobuf, and ffi. That work is ongoing as of this writing; production heaps started with around 160,000 shady objects and now hold roughly 3,000, mostly lingering in google-protobuf and ffi because releases lag behind the merged fixes.

Bringing Back Out-of-Band GC

With tuning knobs mostly exhausted, the reality is that completely removing major marks isn't possible. Some transient objects inevitably get promoted to the old generation and stay there until the next major GC. They are a necessary cost of any Ruby process’s lifecycle.

If we couldn't eliminate major GC, we could at least make it run at better times—outside of request handling. Out-of-Band (OOB) GC was popular pre-Ruby 2.1 and then fell out of favor, mostly because running a full mark after every request is a throughput disaster when those marks take seconds, as ours did.

With the previous work reducing our mark costs drastically, running a full mark every request still wasn't right. But at random intervals—say once per 100 requests—it made sense. The result was dramatic. Major marks during request cycles nearly disappeared:

Where default was the pre-tuning state, gc_tuned_2 was the earlier instrumentation and preallocation work, and gc_tuned_3 adds OOB GC on top of that.

The tail metrics improved hardest. P99.9 and P99.99 time spent in GC dropped roughly tenfold:

Even P99, a less extreme percentile, improved substantially:

Unexpectedly, median requests effectively stopped seeing GC pauses altogether:

The original concern with OOB GC was extra CPU usage. The once-every-100-requests interval was arbitrary, so we tested other frequencies. We settled on an adaptive scheme that starts at one GC per 128 requests and gradually lengthens the interval to one per 512 requests as the process ages. Young processes may still load objects lazily and hold them forever, so they need more frequent old generation cleanups. Those late allocations plateau, meaning an older process can tolerate less frequent marks without regressing tail latency.

Lowering frequency didn't hurt the tail numbers. Median GC time per request returned to ~2ms, which we accept; tail latency was the target. There may be smarter approaches, like predicting imminent major marks based on states such as those exposed by GC.stat, but that path remains unexplored for now.

Rolling Out and Refining

After about three weeks of iterative testing, the team was satisfied enough with the results to apply the new configuration across the entire fleet. Because most experiments ran on 33–50% of production traffic, the three-week tuning effort is visible in global GC time metrics as a gradual step-down.

The tuning work also exposed several longer-term opportunities for the Ruby Infrastructure team at Shopify to improve the GC itself, not just its configuration.

Longer-Term GC Improvements

One avenue under exploration is a new “permanent” generation that would hold objects known to be uncollectable, potentially making major marks far smaller. A major mark currently has to scan the entire old generation; isolating truly permanent objects could reduce that overhead significantly.

Another idea targets the aging process of young objects. The team believes larger API requests can trigger more than three minor marks, which promotes objects that should have remained ephemeral into the old generation. Two possible fixes are on the table:

  • Introducing an “aging factor” so not every minor mark increments an object’s age — see the aging factor pull request.
  • Allowing promotion to the old generation to be entirely disabled once the process has warmed up.

The team is also proposing to expose GC trigger reasons directly via GC.stat to make instrumentation easier.

Whether any of this lands in Ruby 3.3 remains to be seen, but the work demonstrates that even mature runtimes can yield meaningful performance gains when their garbage collection behavior is carefully measured and adjusted.