A Generic Format for a Persistent Cache
Marshal solves a problem that Ruby developers rarely think about: it can serialize almost any object graph with no configuration. That convenience becomes a liability in a large, long-lived codebase, where cached payloads may outlive the classes they reference. When a deploy removes or renames a class, any cache entry containing an instance of that class becomes unreadable—or worse, raises errors at the worst possible moment.
MessagePack offers a different contract. It is a generic binary format with implementations in many languages, and crucially, it knows nothing about Ruby objects unless you explicitly teach it. That limitation is exactly what makes it suitable for a cache that must survive code changes. If MessagePack cannot encode an object, it fails loudly at write time, giving you the opportunity to inspect and decide whether the object belongs in the cache at all.
The Shape of the Payload
At first glance, MessagePack looks like Marshal with different method names: .pack replaces .dump, .unpack replaces .load. The core types are similar—nil, integers, booleans, floats, strings, arrays, and hashes. The difference is in what is absent. MessagePack has no built-in representation for Ruby Object instances or instance variables, which are precisely the constructs that tied Marshal payloads to specific class definitions.
The encoding differences are visible at the byte level. A UTF-8 string "foo" in Marshal includes a TYPE_IVAR wrapper carrying the string's encoding as an instance variable named :E:
MessagePack omits encoding metadata entirely—UTF-8 is assumed—and packs the type tag together with the length:
The result is a payload that is shorter and contains no Ruby-specific annotations. For strings and other simple types, this compactness is a constant win.
Extension Types for the Rest
Real cache payloads contain more than primitives. After an audit of Rails.cache.write calls in Shopify's core monolith, it was clear that the supported types would not cover everything. MessagePack's extension types fill that gap: you register a type code (0 to 127), a class, and a packer/unpacker pair on a MessagePack::Factory.
The Date extension is the simplest example in production. Its packer extracts year, month, and day and packs them into a binary string using the format "s< C C"—the year as a 16-bit signed integer, month and day as 8-bit unsigned integers. The unpacker reverses the process with String#unpack and passes the three values to Date.new:
The encoded form is compact: d603 e607 0909 for July 9, 2019, where d603 is the extension type marker and the remaining bytes carry the date components. Extension types make it possible to store any object in a form that is self-describing, without dragging in class references.
Failing on Purpose
The decision to migrate was not just about format efficiency. The core problem was that Marshal was happily serializing objects whose classes changed during a deploy, leaving cache readers in old code unable to instantiate them. The fix was to prevent those objects from entering the cache at all.
MessagePack's default behavior gives you that guarantee for free. Attempting to encode an unregistered object—say, a plain Object nested inside a hash—raises a NoMethodError with a message like:
NoMethodError: undefined method `to_msgpack' for <#Object:0x...>
The exception surfaces at write time and includes a reference to the offending object. That is recoverable and actionable. Logging the exception message and the object's class tells you whether you need a new extension type or whether you are caching something that should never be cached.
Running Both Formats
The migration was deliberately incremental. Marshal and MessagePack coexisted for roughly six months while extension types were added and cache contents were audited. The write path had three branches:
- Try to encode with the MessagePack factory, using the registered extension types.
- On success, prepend a version byte identifying the extension types used, and store the MessagePack payload.
- On a
NoMethodError, log details about the failed object and fall back to Marshal.
The fallback branch was the primary source of migration intelligence. Logging and StatsD metrics captured the class of every object that MessagePack refused to encode, driving new extension type creation and exposing cache entries that never should have been written. A Marshal payload could be identified when reading by its 0408 prefix; anything else was MessagePack.
The initial extension type set was small, assembled from types already registered in the monolith for earlier MessagePack work:
Symbol, available in the messagepack-ruby gem and enabled explicitlyTimeDateTimeDateBigDecimal
That set covered basic values but not the records that a Rails application serializes heavily. An extension type for ActiveRecord::Base was the next requirement, and the rescue path had already identified that gap from real traffic.
A Compact Representation
Records are defined by their attributes, but caching only those attributes misses a key part of the picture: associations. Marshal encodes the full set of loaded associations along with the record, so that no extra database queries are needed on deserialization. An extension type that caches only attribute values would require a fresh query to refetch those associations later, which is far less efficient.
To handle this, we built a serializer called ActiveRecordCoder. The encoding algorithm builds a tree where each association is represented by its name (for example :comments or :post) and each record is represented by its unique index in a structure called an Instance Tracker. Untracked records are traversed recursively through their association network; records already seen are encoded by their index alone.
This yields a compact tree, and when the post with two comments from our earlier example is encoded, the resulting MessagePack payload is around 300 bytes. Compare that to the 1,600-byte Marshal payload for a post with no associations, and the savings are clear. Marshal's payload for the post with its two comments is over 4,000 bytes — making our combination of ActiveRecordCoder and MessagePack 13 times more space efficient for that case.
The efficiency gain showed up immediately in our data analytics. Rails cache memcached fill percent dropped after the switch. For simple payloads like booleans and integers the change was modest, but for complex objects such as records the impact was substantial: total cache usage dropped by over 25 percent.
Rails cache memcached fill percent versus time
Graceful Degradation
ActiveRecordCoder includes the names of record classes and association names in encoded payloads, which might look like a regression to the Marshal behavior that caused problems initially. There are two key differences, though.
First, because we control the encoding process, we decide how to handle missing class or association names: we rescue the error and raise a more specific one. Second, because this is a cache and not a persistent store, we can afford to occasionally drop a stale payload. So when we see an exception for a missing class or association name, we treat the cache fetch as a miss:
The practical effect is that during a deploy where class or association names change, affected cache payloads are invalidated and the cache refills them. The cache may effectively be disabled for those keys during the deploy window, but it returns to normal operation once the new code is fully rolled out. That's a reasonable tradeoff and far more graceful than what Marshal offers.
Subclass Gotchas
As we prepared to ship the first migration step, tests on CI started failing on hash-valued cache payloads. The culprit was HashWithIndifferentAccess, an ActiveSupport subclass of Hash that allows symbols and strings to work interchangeably as keys. Marshal handles such subclasses out of the box. MessagePack does not — it serializes a HashWithIndifferentAccess back as a plain Hash:
MessagePack doesn't raise an error because HashWithIndifferentAccess is a subclass of Hash, which it does support. That silent fallback is dangerous; we would have preferred an exception so we could fall back to Marshal. Tests caught the issue before it reached production.
Interestingly, defining an extension type for HashWithIndifferentAccess didn't fix the problem—MessagePack ignored it entirely. The issue was in msgpack-ruby: extension type handling didn't trigger for subclasses of core types like Hash. We submitted a pull request (PR) to fix it, and as of version 1.4.3, msgpack-ruby supports extension types for Hash, Array, String, and Regex.
The Last Five Percent
With that fix, we shipped the first phase of the migration. MessagePack successfully serialized 95 percent of payloads immediately. As expected, the remaining 5 percent was the hard part. We added extension types for commonly cached classes like ActiveSupport::TimeWithZone and Set, but couldn't reach 100 percent—too many different objects were still being cached with Marshal.
Defining a new extension type for every case wasn't viable. Shopify has thousands of developers, and MessagePack caps extension types at 128. Instead, we adopted a catchall type for Object, the parent of most Ruby objects. The Object extension type looks for two methods on each instance: as_pack as a serializer and a class method from_pack as a deserializer. If both exist, the object is packable:
As with ActiveRecordCoder, this relies on encoding class names, which is safe only because we handle name changes as cache misses. This approach wouldn't work for a persistent store.
That catchall worked well, but many cached objects followed a similar pattern: Structs or T::Structs (Sorbet typed structs), all defined by a set of attributes. We extracted the packable logic into a module that, when included, makes a struct packable automatically:
The serialized data includes an extra digest value (26450) capturing the names of the struct's attributes. A changed digest signals to the Object deserializer that attribute names have changed, and the cache treats the data as stale and regenerates it:
By including this module—or the analogous one for T::Struct—developers can cache structs robustly against future changes. With these modules, we quickly migrated the remaining types. Once logs confirmed that Marshal was no longer serializing any payloads, we removed it from the cache entirely. We now cache exclusively with MessagePack.
A Safer Cache by Default
With MessagePack as the serialization format, the cache in the core monolith became safe by default—not safe under special conditions, not safe most of the time, but unconditionally safe. For a platform of Shopify's scale and complexity, the importance of this shift to stability and scalability is difficult to overstate.
For developers, a safe cache means one less unexpected failure mode when shipping refactors. That peace of mind makes large, challenging refactors more likely to proceed, which improves the overall quality and long-term maintainability of the codebase.
Paquito: The Path Forward
If this approach sounds worth trying, most of the work from this project has been extracted into the Shopify/paquito gem. A migration away from Marshal will never be trivial, but Paquito incorporates the lessons learned from Shopify's own experience. It is intended to help others move toward a safer cache.
