Safely Landing New Features With Beta Flags

Shopify deploys code multiple times a day, which means every change carries some risk to the million-plus merchants using the platform. Beta flags are one mechanism the company uses to de-risk feature development. They reduce the blast radius of a change by rolling it out to a percentage of users, let teams decide precisely when a change becomes active in production (rather than at deploy time), allow instantaneous rollback, and enable developers to test otherwise-inactive code paths in production.

The Core Building Blocks

“Beta” is an overloaded term in software. Before building anything, it helps to define a few primitives.

Subject is the entity you want to control features against. For multi-tenant SaaS this is usually the tenant model—for Shopify, typically a shop. A polymorphic design lets you apply the same mechanisms to multiple kinds of entities later.

BetaIdentifier is a simple string naming the feature, like multi_location. If you go this route rather than using auto-incrementing integers, watch out for case-sensitivity and for accidentally reusing the string. You can attach metadata—description, ownership, notification channels, expected behavior when on or off—for internal tooling and documentation.

BetaFlag is the smallest piece of state: data associated with a single Subject, following a “Subject has_many BetaFlag” relationship. Each BetaFlag holds a BetaIdentifier plus timestamps. With a BetaFlag you can ask whether a given Subject has the flag (feature on), and list all Subjects that do. This gives you explicit per-Subject toggling.

BetaRollout is the percentage-based counterpart, and it isn't tied to any individual Subject. It holds a beta_name (a BetaIdentifier), a percentage_rollout integer from 0 to 100, and a method that decides whether a given Subject counts as rolled out. If no BetaRollout record exists for a feature, the feature is off.

One performant way to implement BetaRollout#enabled? is to compute a digest of the two identifiers, convert it to an integer, and take it modulo 100. A few properties make this worthwhile:

  • As the rollout percentage increases, each step hits a different—but growing—set of Subjects.
  • The modulo result stays static as the percentage changes, so a user who saw the feature at 10% still sees it at 20%.
  • Features don't flicker on and off for users as rollout widens, and the same subset of Subjects isn't repeatedly affected across independent rollouts.

Combining Explicit Flags and Percentage Rollouts

With a BetaIdentifier defined, you have two levers: apply a BetaFlag directly to a specific Subject, or create a BetaRollout at a given percentage. Whether a feature is enabled for a given Subject is then “does the subject have the flag explicitly (BetaFlag) or implicitly (BetaRollout)?” That unified view is the Beta.

Rolling back is straightforward in one case and painful in another. If you only used the percentage mechanism, set the BetaRollout percentage to 0 and you're done—though anyone with an explicit BetaFlag still sees the feature. If you manually applied flags to thousands of Subjects, rollback means writing a maintenance task to update the database, or possibly not being able to revert at all in the worst case. That's a real problem during an incident.

The fix is to avoid referring to the Beta primitives directly in your feature code. Add another abstraction layer on top. Call it a Feature: a higher-level construct that lets developers:

  • Apply a feature directly to one Subject or thousands, for testing or for rollouts that can't be percentage-based.
  • Apply a feature to a random percentage sample of all other Subjects, for the standard gradual rollout.
  • Apply an escape hatch—say, my-cool-new-feature-opt-out—to specific Subjects that hit problems but don't warrant a full rollback.
  • Apply a kill switch by rolling the opt-out flag out to 100%, instantly disabling a feature for everyone even when thousands of BetaFlags can't be removed in time.

That last option is the runaway-train safety net: an instant, global halt applied through the same mechanisms you already use for regular rollouts.

Why the Extra Layer Matters

Simple yes/no qualifications on a beta often evolve. Requirements change, eligibility rules get more complex, and the set of conditions for enabling a feature grows. Coding against a Feature abstraction means changing one method when that happens, instead of hunting down dozens of interspersed Beta.enabled?(...) calls. The same reasoning applies to exposing functionality over an API: if N clients still refer directly to the primitives, you can't update all of them at once. A stable, higher-level abstraction keeps that flexibility open as your beta program matures.

Pitfalls to Plan For

Beta-based development is a powerful tool, but it is not without its sharp edges. Being aware of the common failure modes can save you from painful rollbacks and lingering technical debt.

Data Compatibility and Rollback

The primary value of this pattern is the ability to switch code paths quickly. However, if code path A generates data that is incompatible with code path B, a rollback will create problems unless you have designed the rollback experience from the start. Switching paths is easy; reconciling the data left behind by the previous path is far harder. Always consider the state of your data when you flip the switch in either direction.

Irreversible Features

Some features are not good candidates for this approach. If a change impacts your underlying data model, or if migrating between paths is not feasible, a rollback may leave the system in a nonsensical state. Similarly, once a user depends on a new feature or business concept, removing it may be more disruptive than leaving it in place. In such cases, it might be wiser to iterate on the feature rather than trying to gate it.

If you do wish to prevent new users from seeing a feature while still supporting existing users, the primitives can handle this. By manually applying a BetaFlag to a specific set of Subjects while rolling the BetaRollout back to 0%, you can grant access to a select group while turning off the rollout entirely for everyone else.

Identifier Hygiene

If you are using a magic string as a handle for your BetaIdentifier, treat it as a permanent resource. Reusing a name that exists in your database with stale flags or rollout settings can cause the feature to activate unexpectedly on certain subjects the moment you deploy. This is a rare issue in practice, but it highlights the need for proper deprecation and deletion of old beta records.

Operational Discipline

Shipping the code for a feature is only the first step; the feature remains dormant until the flags are applied. Because a percentage-based rollout essentially changes the running code path for a subset of users, Shopify now treats these changes with the same seriousness as a deployment. When someone adjusts a rollout percentage, the change is announced in the #operations chat channel along with the reasoning. This provides a crucial data point if exceptions spike or metrics dip, eliminating the guesswork of operating in the dark.

Applying a beta flag directly to a subject also triggers a Slack notification to the teams that built the feature. This alerting helps prevent erroneous applications and keeps feature owners informed of "opt-out" beta requests, allowing them to steward the feature more effectively.

Testing in a Beta World

The standard approach—write unit and integration tests that enable the feature and verify the new path—has a blind spot. While your new tests cover the beta path, the rest of your suite is still exercising the default, non-beta code path. If a feature sits at 100% for months, nearly your entire test suite is testing code that is no longer live in production. This often surfaces when the beta flags are finally removed, causing hundreds of seemingly unrelated tests to fail because they were never updated for the new code path.

Running the entire suite twice—once with the beta on and once with it off—leads to a combinatorial explosion of permutations and is a waste of CI time. For most small features, the extra effort of fixing tests after a rollout is minor. For particularly complex features, running whole files (e.g., a specific controller spec) against both paths can provide an extra degree of confidence.

As a practical tip, hardcoding the feature flag to true on your branch and observing which tests fail can be a quick way to uncover missed edge cases in your implementation.

Cleanup is Part of the Job

If a feature has been at 100% without issues, it is tempting to leave the beta flags in place just in case. Keeping them for a few months is a reasonable safeguard. However, leaving them indefinitely turns the pre-beta code path into dead code. As teams move on to other projects, that old path becomes maintenance debt that eventually needs to be audited and removed.

The Takeaway

These primitives and patterns have enabled Shopify to ship amost everything—from large feature rollouts to performance tweaks—with confidence. Having a mechanism to control software after it has been deployed is a powerful advantage. However, this system is not a silver bullet; the implementation requires careful thought about caching and other performance considerations. These concepts are a starting point for defining what "beta" means in your own environment and empowering your developers to release safely.