Locking Down Gem Sources Across Shopify’s Ruby Fleet

Shopify’s bug bounty program regularly surfaces real-world attack vectors. One report highlighted a dependency confusion vulnerability that could let an attacker execute code in local development, CI/CD, and production environments. The root cause: package resolution ambiguity. If a public gem shares a name with an internal one and has a higher version number, Bundler could resolve to the external package.

The Ruby Conventions team at Shopify took on the remediation across more than 600 applications. The fix wasn't just a matter of upgrading Bundler; it required updating the Gemfile.lock format across the entire fleet without introducing regressions.

The Problem With the Bundler Fix

Bundler addressed the underlying issue by introducing a new Gemfile.lock file format that pins each gem to an explicit source. This format is generated on a fresh install or an update. The complication: generating the new lockfile meant a full dependency update. That requires vetting every change for behavioral regressions, a manual process too slow for hundreds of repositories.

Measuring the Scope

Before planning a migration, the team needed data. They set up a cron job in CI to extract Bundler version information from all repositories into Shopify’s data lake. The result: roughly 600 Ruby applications were running a vulnerable Bundler version. This metric also provided a way to track migration progress and decouple the solution from the end goal.

Prototyping an Automated Migration

Manual migration of hundreds of lockfiles wasn't feasible. The team needed a way to update the lockfile format without changing dependency versions. They experimented with a Bundler plugin—a gem that extends Bundler's functionality. The plugin's logic was straightforward:

  1. Initialize the gem specification from the existing Gemfile.lock file, capturing names, versions, and remotes.
  2. Update the file to the new lockfile format—this does update all gems, so the plugin limits changes to patch versions only.
  3. Rewrite the updated lockfile, replacing the new versions with the originals from the old file.

This approach wasn't flawless, but it was functional. It unblocked the path to a large-scale automated rollout.

Migrating Hundreds of Repositories

Rather than pre-emptively solving every possible edge case, the team started small. They ran the plugin on a handful of non-merchant-facing applications. After those succeeded, they expanded to a larger batch of non-critical repositories. This is where inconsistencies surfaced—build setups, Ruby versions, and other configurations varied significantly across the fleet.

Collaboration with infrastructure teams was essential. When tooling didn't support the latest Bundler version, the team investigated the issue, attempted a fix, and then shared the full context with the relevant team. This approach made it straightforward for others to help.

One notable external collaboration involved Heroku’s Ruby buildpack, which didn't support the required Bundler version. Working with the Heroku Buildpack team led to a new release that included the updated Bundler, benefiting not just Shopify but the wider Ruby community.

Communication with application owners and a clear deprecation deadline for the old Bundler version were critical. Teams needed to understand the security impact to prioritize the work.

The migration plugin originally ran locally, but this didn't scale. Managing Ruby versions, parallel execution, and failures became unwieldy. The solution was a CLI tool built on top of Shopify's CI system. It set up an isolated environment per repository, ran the migration commands, and opened a pull request with the changes. This design eliminated configuration issues and allowed parallel execution, which dramatically sped up the process and made failures easier to track and recover.

Addressing Regressions Preemptively

The team knew lockfiles could regress to the older, vulnerable format. They chose not to build prevention tooling upfront. Instead, they monitored for regressions during the migration and investigated each case manually. No deeper systemic issues were found, so the migration continued.

Findings from these investigations were shared with the Bundler team. Bundler was subsequently enhanced to prevent these specific regression cases, eliminating the need for Shopify to build custom tooling. The only remaining requirement was ensuring all applications used the correct Bundler version.

Because migrations were staggered to maintain steady progress, most applications were not on the fully patched Bundler version immediately. Once the migration tooling was battle-tested and configuration issues were resolved, the team pushed all remaining applications to the latest Bundler version in under a day.

Two additional preventative measures were implemented:

  • Local environment tooling now defaults to the recommended Bundler version, protecting individual developer machines from malicious gem execution.
  • CI now fails on encountering an outdated Bundler version, blocking any code change that could reintroduce the vulnerability.

The iterative approach—starting small, discovering edge cases through real-world execution, and continuously improving tooling—turned a potentially massive migration into a manageable, de-risked deployment.

Working With the Bundler Maintainers

Shopify’s philosophy is that contributions to open source should go beyond pull requests. Sharing the context behind investigative work, surfacing problems, and testing prototypes are all valuable ways to give back. With that in mind, the team took its internal fix for the dependency confusion vulnerability and proposed changes in Bundler that would allow the tool to update Gemfile.lock without upgrading gems in the process.

That specific proposal wasn’t merged, but it led to a conversation with maintainers that produced an alternative approach, shipped in Bundler 2.2.21. Shopify then helped test that implementation across its applications, working through edge cases to reduce the burden on the wider community.

Preventing Lockfile Regressions

Testing surfaced another issue: developers running an insecure Bundler version could accidentally revert to the old lockfile format. The then-current Bundler would still resolve an older Gemfile.lock on bundle install, making it easy to regress. Shopify built a prototype to block that path, which flagged the problem for the Bundler maintainers. Their subsequent release, version 2.2.22, prevents the regression and hardens the tool for everyone.

What the Effort Achieved

The original goal—fixing dependency confusion in every Ruby project at Shopify—was met. The success came from an iterative approach that accommodated changing circumstances while maintaining steady progress. Along the way, the team built tooling for large-scale migration that has since proven useful for other work. They also aggregated Bundler version data across Shopify’s Ruby projects, creating a basis for future adoption decisions.

The collaboration with the Bundler team improved base functionality, with Shopify’s scale exposing edge cases, bugs, and areas for enhancement that ultimately make the tool better for the entire Ruby community.