Why GitHub wraps risky changes in flags

GitHub runs deployments around the clock and needs the service to stay up throughout. A bad deployment today means a rollback, and during that window users can be affected and other deployments sit blocked. The longer rollback takes, the wider the impact. Feature flags shrink that danger: any potentially risky change ships behind a flag, the deployment completes, and then the flag is turned on for everyone or for a percentage of actors. If something goes wrong, the flag can be killed in seconds without touching the deployment. That matters because the problems flags catch are often not code crashes but subtle regressions—slower database queries, unexpected behavior in existing logic, or writes that start inserting invalid values into new columns.

Flags also change how GitHub builds new features. Instead of long-lived feature branches, engineers merge small batches and keep the unfinished work hidden behind a flag that only the project team has enabled. Small batches are easier to review in pull requests and less likely to go wrong in production, and avoiding long-lived branches cuts merge conflicts with other work in flight.

That approach requires some planning up front, usually around data model changes that other pieces depend on. Two patterns help unblock work:

  • Create a main “spike” pull request containing the data model changes, then keep building UI, API and background jobs on top of it while extracting smaller reviewable pull requests from that work.
  • Or open an initial pull request and branch new work off that branch before it merges, rebasing the dependent branches if the first one changes.

Review latency can still stall progress, so teams use a few tricks: keep coding on top of the unreviewed branch or spike, designate a first responder per team to review pull requests quickly, and defer non-critical feedback on a ready-to-deploy pull request to follow-up PRs.

How flags are toggled during development and testing

Feature flags are controllable at every stage of the pipeline:

  • Locally, from the command line in development environments.
  • In automated tests, from within the test code.
  • In CI, via two builds—one with all flags disabled and one with all enabled—so most code paths get exercised.
  • In production, via the query string of a request.

Shipping strategies beyond a simple on/off

GitHub uses several targeting modes for production flags, managed through a web UI available to most engineers:

  • Individual actors: Flag specific users or organizations, useful for employees building a feature, customers experimenting, or users with a bug being investigated.
  • Staff shipping: Before a public release, enable the flag for all GitHub staff, post internally, and collect feedback in an internal issue.
  • Early access groups: For features that affect OSS maintainer workflows, test with a small beta group first, then interview them to validate assumptions.
  • Percentage of actors: A specified share of actors gets the flag, and once an actor is flagged they stay flagged unless the flag is rolled back. Changes to the percentage are announced in the deployments Slack channel.
  • Dark shipping: Enables a flag for a percentage of calls rather than actors, so the same actor might see the feature on one request and not the next. This is reserved for internal changes like query performance work, not user-visible features.

The UI supports creating, deleting and changing the shipping status of flags, with a full history of who changed what and when, plus metadata like the owning team.

What counts as an actor

Actors are not just users. Flags can target users, organizations, teams, enterprises, repositories, or GitHub Apps, and the code checks the appropriate actor for the change:

  • Purely visual changes, like dark mode or navigation tweaks, flag the current logged-in user.
  • Changes to data storage flag the repository, keeping writes consistent for everyone using that repo rather than splitting by user.
  • API changes flag GitHub Apps.
  • Risky or unusual changes sometimes get custom, context-specific actor types.

Checks can also combine conditions. When staff shipping a feature that must not leak publicly, for instance, the flag may require both that the actor is an employee and that the repository is private, so an employee’s public repos don’t expose the new feature.

The real cost of flags

Flags are not free. Every check loads the flag’s metadata—stored in MySQL and cached in memcached—plus the actor list. GitHub is considering keeping metadata and actor lists fully in memory to cut that overhead. Large flags, like the GitHub Actions rollout that added tens of thousands of users weekly, are an exception: while not fully enabled they require a per-actor MySQL query to test membership.

There is also the cleanup burden. Once a flag is fully rolled out, the codebase is littered with dead branches and obsolete tests that must be deleted manually. GitHub is automating that with a script that can run locally or via a manual workflow dispatch. The script finds flag usages with regular expressions over git grep output, then rewrites matched Ruby files with rubocop-ast, deleting if/else blocks, simplifying boolean expressions and reindenting. In tests it removes the lines that enable the flag; for tests that assert behavior when the flag is disabled, it leaves a comment and forces a failure so an engineer updates instead of blindly deleting the test. When dispatched as a workflow, the script also creates a branch and a pull request, and since GitHub leans on CODEOWNERS, the right reviewers are attached automatically. The next step is running the script automatically once a flag is considered stale.

The trade-off is worth it: flags lower deployment risk, enable small-batch development, and let teams ship faster with higher confidence.