Code Ranges: A Deeper Look at Ruby Strings - Shopify

Encoding-Aware Strings and the Cost of Flexibility

Ruby's string handling stands apart from most languages. Strings are mutable by default, encoding-aware, and the core library ships with over 100 encodings that can be applied to any string. This flexibility lets Ruby support legacy applications and niche platforms, but it comes at a price: the runtime must consider encodings in nearly every string operation.

When appending two strings, the runtime checks encoding compatibility. Other operations need to validate data against the attached encoding or locate character and grapheme boundaries. The efficiency of these operations depends heavily on the encoding in use. A string containing only valid ASCII characters is simple to handle—each character is a single byte, so methods like String#[], String#chr, and String#downcase are fast. Fixed-width encodings allow efficient character-offset calculations. UTF-8, the default internal encoding, is variable-width (1–4 bytes per character), which complicates operations: determining character counts or offsets requires scanning the entire string. However, UTF-8's backwards compatibility with ASCII means that a pure-ASCII UTF-8 string behaves like a string with the simpler ASCII encoding—provided the runtime knows it can optimize.

Validating a string's bytes against its encoding is an O(n) operation. Other languages avoid this cost by validating once at creation, during compilation, or by relying on immutability. Ruby strings are none of those things, so the runtime caches validation results in a field known as a code range.

The Four Code Range States

Every Ruby string carries one of four code range values:

• ENC_CODERANGE_UNKNOWN
• ENC_CODERANGE_7BIT
• ENC_CODERANGE_VALID
• ENC_CODERANGE_BROKEN

The code range is an implementation detail—there is no direct API to query it—but its value affects operation performance. A few public String methods let applications infer the code range from the output they return.

Table 1: Mapping of internal code range values to public Ruby methods.

The * marker in the table indicates that code ranges are usually calculated lazily. When you request a property that depends on the code range—like a string's character length or validity—the runtime computes and caches it on demand. Until then, a string may carry ENC_CODERANGE_UNKNOWN indefinitely.

Given its dual nature as both internal detail and performance factor, every major Ruby implementation tracks a code range on each string. Working on Ruby internals or native string extensions will inevitably involve managing this value.

Representation and Semantics

MRI stores the code range as a two-bit mask in the object header. As with any enum, the four values are mutually exclusive. That exclusivity matters: a string with an ASCII-compatible encoding containing only ASCII characters is logically valid, but it will always be reported as ENC_CODERANGE_7BIT, never ENC_CODERANGE_VALID. When reading the code range in C code, use the ENC_CODERANGE(obj) macro and compare against individual constants:

if (cr == ENC_CODERANGE_7BIT) { ... }

A bitmask mindset will lead to subtle bugs. The compact two-bit representation means a hypothetical string flagged as both ENC_CODERANGE_7BIT and ENC_CODERANGE_VALID would appear as ENC_CODERANGE_BROKEN. Combining masks with logical OR for a branch, such as cr & (ENC_CODERANGE_7BIT | ENC_CODERANGE_VALID), inadvertently includes ENC_CODERANGE_BROKEN strings. The space savings in the object header come at the cost of clarity.

JRuby mirrors MRI's compact int representation. TruffleRuby uses a full enum stored in the object's shape—more memory, but immune to bitmask confusion.

When Code Ranges Change

A string's code range depends on both its byte sequence and its encoding. Any operation that alters either can invalidate the cached value. The conservative approach—rescanning the entire string—is what the runtime tries hard to avoid.

MRI uses two strategies to minimize unnecessary scans. The first is lazy invalidation: when an operation changes a string in a way that could affect its code range, MRI resets the value to ENC_CODERANGE_UNKNOWN. The actual scan is deferred until some later operation actually needs the resolved code range. The VM does compute eagerly in one case: when lexing source files, MRI already examines every byte of a string literal, so discovering and caching the code range is nearly free.

The second strategy is reasoning about operations on known code range values. Operations on ENC_CODERANGE_7BIT strings rarely require a fresh scan. Substrings, case changes, and whitespace stripping all preserve the fact that every byte remains within 0x00–0x7f, so the result is guaranteed to also be ENC_CODERANGE_7BIT.

Listing 2: Changing the case of a string with an ENC_CODERANGE_7BIT code range always results in a string with ENC_CODERANGE_7BIT.

But the code range alone sometimes lacks context needed for optimizations. MRI tracks an additional property, "single-byte optimizable," for strings that are either ENC_CODERANGE_7BIT or whose encoding uses one-byte characters only, like ASCII-8BIT/BINARY. For such strings, operations like String#reverse are safe: reversing bytes can't change their meaning because byte and character boundaries coincide.

Operations on ENC_CODERANGE_VALID strings are trickier. The result could be ENC_CODERANGE_7BIT (if the source encoding is ASCII-compatible and the operation eliminates all non-ASCII characters) or ENC_CODERANGE_VALID. A full scan may be unavoidable to determine which. Notably, String#setbyte can push a string into ENC_CODERANGE_BROKEN, although most string operations are well-defined enough to avoid this outcome.

Listing 3: Case-changing a string with ENC_CODERANGE_VALID might yield a different code range.

Because a string with ENC_CODERANGE_UNKNOWN might not resolve its code range during an operation—say, String#reverse on a BINARY string—the result can also carry ENC_CODERANGE_UNKNOWN. An ASCII-only string may sit in this state until some later operation forces a full scan. This is the inherent trade-off of lazy computation. End users notice no difference because the value is always accurate before being used. The distinction matters for native extension authors, runtime developers, and anyone profiling Ruby applications: a code range can be accurate, approximate, or simply deferred.

Code Ranges in TruffleRuby: Eager Computation and Ropes

TruffleRuby takes a different approach to code ranges than MRI or JRuby. Instead of leaving strings with an ENC_CODERANGE_UNKNOWN value until the range is needed, TruffleRuby computes code ranges eagerly so that no string is ever in an unknown state. The cost is that some code range values may be calculated and never used, but the benefit is that string operations never have to pause to compute a code range on demand. In many cases, TruffleRuby can also derive the code range of a result string without scanning its bytes at all.

This eager strategy looks potentially wasteful, but it pays off because TruffleRuby reuses string data heavily. Strings are represented as ropes: a tree structure whose leaves are conventional C-style strings and whose interior nodes represent operations that join other ropes together. (TruffleRuby's rope implementation was promoted to a top-level library in the Truffle family and is now shared by other languages in the GraalVM distribution, so references to "rope" in the codebase may be sparse.) A Ruby string points to a rope, and the rope holds the critical string data.

On a concatenation, for example, TruffleRuby creates a "concat rope" whose children are the two strings being joined, rather than allocating a fresh buffer and copying bytes. The result string is updated to reference the new concat rope. That concat rope stores no byte data of its own, but it does carry a code range value that is trivial to derive since each child rope already has a code range and an associated encoding object.

Rope metadata is immutable, so reading a rope's code range is no more expensive than a field access. TruffleRuby uses this property to make ropes guards in inline caches for its JIT compiler, and it can specialize string operations based on the code ranges of argument strings. Because most Ruby programs never encounter ENC_CODERANGE_BROKEN strings, the JIT can eliminate code paths dealing with that range; if a broken string does show up at runtime, the JIT deoptimizes and handles it on a slow path, preserving Ruby's full semantics. The same logic applies to encodings: although Ruby supports over 100 out of the box, TruffleRuby's JIT optimizes for the small subset an application actually uses.

Strings Behind the Scenes

String performance discussions usually focus on web template rendering or text processing, but strings are also pervasive inside the Ruby runtime itself. Every symbol and regular expression carries an associated string that gets consulted frequently. More interesting is Ruby's metaprogramming: strings can access instance variables, look up methods, send messages to objects, evaluate code, and more. That means improvements—or regressions—in string performance ripple through the whole VM.

Code ranges are not the whole story for fast metaprogramming, though. They help disqualify strings quickly when the range is known not to match—for example, rejecting ENC_CODERANGE_BROKEN strings outright. In principle, a check for ASCII-only identifiers could dismiss ENC_CODERANGE_VALID strings when only ENC_CODERANGE_7BIT is acceptable, though MRI does not currently do this. After the code range check passes, the string still needs to be matched against an identifier. TruffleRuby handles that by interning its immutable ropes, so comparison can be done by reference. MRI and JRuby may need a linear scan of the string data during interning, and the picture gets more complicated depending on whether the string is dynamically generated or a frozen literal. (For a deeper look, Chris Seaton has published a paper on making metaprogramming fast in Ruby, and Kevin Menard has given a related talk at RubyKaigi.)

What Code Ranges Are Really For

Ruby exposes many features that are tricky to optimize while still giving developers a lot of expressive power. Code ranges let the VM avoid repeated work and tailor operations to individual strings, steering clear of slow paths when special functionality is not needed. That benefit has historically been most visible in the interpreter. With a JIT that can deoptimize, like TruffleRuby's, code ranges also let the compiler remove generated code for the kinds of strings a given application and the VM itself actually use.

Understanding code ranges also helps with debugging, both for correctness and performance. A code range is ultimately a cache—and like any cache, it can hold a stale value. Such bugs inside the Ruby VM itself are rare, but native extensions that manipulate strings sometimes forget to update a string's code range. Knowing how code ranges work makes Ruby's string handling less of a black box when issues do appear.