Search Indexes No Longer a Single Point of Failure in GHES
Search underpins far more of GitHub than the obvious search bars. Issue filters, pull request counts, releases pages, and project views all depend on it. In GitHub Enterprise Server, that dependency made search indexes a delicate part of administration: a misordered upgrade or maintenance step could corrupt indexes, lock them, and stall the whole appliance. The root cause was how Elasticsearch was integrated into High Availability (HA) deployments, and fixing it has taken a multi-year engineering effort.
From Clustered Search to Independent Nodes
HA GitHub Enterprise Server follows a leader/follower pattern: the primary node handles all writes and traffic, while replica nodes stay synchronized and stand by to take over. Historically, Elasticsearch couldn't support this directly. Instead, GitHub engineering ran a single Elasticsearch cluster spanning both the primary and replica nodes. That made data replication straightforward and gave each node fast local access to search requests, but it created serious operational hazards.
In that clustered mode, Elasticsearch could promote a primary shard—the component that receives and validates writes—onto a replica node. If an administrator then took that replica down for maintenance, the whole system could deadlock: the replica would wait for Elasticsearch to be healthy before starting, while Elasticsearch couldn't become healthy until the replica rejoined. GitHub engineers tried to stabilize this across several releases by adding health checks and corrective processes, and even attempted a custom "search mirroring" system. But those efforts ran into the difficulty of implementing consistent database replication in-house.
Elasticsearch's Native Replication Changes the Model
The breakthrough came from Elasticsearch's Cross Cluster Replication (CCR) feature. Instead of one cluster spread across servers, GHES now uses multiple single-node Elasticsearch clusters—one per Enterprise server instance. Each instance operates independently, and CCR handles sharing index data between them.
CCR replicates data only after it has been durably persisted to Lucene segments, Elasticsearch's underlying storage. This guarantees that only confirmed writes are copied to followers. With this architecture, the leader/follower pattern that GHES already uses maps cleanly onto Elasticsearch's own capabilities, and critical data can no longer end up stranded on read-only nodes.
Bootstrap and Lifecycle Workflows
Adopting CCR required more than just flipping a switch. Elasticsearch's auto-follow API only governs indexes created after a policy exists, but HA installations already have long-lived indexes. GHES needed a bootstrap step that first attaches followers to existing indexes, then enables auto-follow for future ones. GitHub also had to build custom workflows for failover, index deletion, and upgrades, since Elasticsearch only handles document replication—the rest of the index lifecycle remains GHES's responsibility.
Enabling CCR Mode
CCR mode is available starting with GitHub Enterprise Server 3.19.1. To use it, contact [email protected] to obtain a license that enables the new HA mode. After downloading the license, set ghe-config app.elasticsearch.ccr true and then run config-apply or upgrade to 3.19.1.
On restart, Elasticsearch migrates the installation to the new replication method. All data consolidates onto the primary nodes, cross-node clustering is broken, and replication restarts over CCR. The migration time depends on the size of the instance. The new method is optional for now, but GitHub plans to make it the default over the next two years, so administrators are encouraged to test it and provide feedback early.
