Adding Persistence for Crash Recovery
Part 2 of this series focused on network partitions — the scenario where servers become isolated from the rest of the cluster. But a real failure mode for distributed systems is also a server crashing and restarting. From the outside, a crash looks like a partition, but on the crashed server itself, all volatile in-memory state is lost on restart.
The Raft paper's Figure 2 marks which state must be persistent: it must be written and flushed to non-volatile storage before the server issues its next RPC or replies to an ongoing one. The set of state that requires persistence is deliberately small:
currentTerm— the latest term this server has observedvotedFor— the peer ID for whom this server voted in the latest termlog— the Raft log entries
Notably, commitIndex and lastApplied remain volatile. After a reboot, Raft can re-derive commitIndex from persistent state alone: once a leader has committed an entry, everything before it is also committed, and the next AppendEntries from the current leader tells a recovering follower the correct value. lastApplied resets to zero on restart, assuming the application (e.g., a key/value database) keeps no persistent state of its own and needs to be rebuilt by replaying the log. Snapshot-based log compaction addresses the inefficiency of full replay, but it's out of scope for this series.
Delivery Semantics and the Storage Interface
With crashes in the picture, a client may receive the same command more than once — for example, when a crashed server replays its log on restart. Raft therefore offers at-least-once delivery semantics. In practice, commands should carry unique IDs, and clients should ignore duplicates; section 8 of the Raft paper covers this in more detail.
To implement persistence, the code introduces a Storage interface:
type Storage interface {
Set(key string, value []byte)
Get(key string) ([]byte, bool)
// HasData returns true iff any Sets were made on this Storage.
HasData() bool
}
This is conceptually a persistent map from string keys to byte slices.
Save and restore
The ConsensusModule constructor now accepts a Storage argument and immediately attempts to restore existing state:
if cm.storage.HasData() {
cm.restoreFromStorage(cm.storage)
}
restoreFromStorage deserializes the persistent state variables using Go's encoding/gob:
func (cm *ConsensusModule) restoreFromStorage(storage Storage) {
if termData, found := cm.storage.Get("currentTerm"); found {
d := gob.NewDecoder(bytes.NewBuffer(termData))
if err := d.Decode(&cm.currentTerm); err != nil {
log.Fatal(err)
}
} else {
log.Fatal("currentTerm not found in storage")
}
if votedData, found := cm.storage.Get("votedFor"); found {
d := gob.NewDecoder(bytes.NewBuffer(votedData))
if err := d.Decode(&cm.votedFor); err != nil {
log.Fatal(err)
}
} else {
log.Fatal("votedFor not found in storage")
}
if logData, found := cm.storage.Get("log"); found {
d := gob.NewDecoder(bytes.NewBuffer(logData))
if err := d.Decode(&cm.log); err != nil {
log.Fatal(err)
}
} else {
log.Fatal("log not found in storage")
}
}
The counterpart persistToStorage encodes and writes the same state:
func (cm *ConsensusModule) persistToStorage() {
var termData bytes.Buffer
if err := gob.NewEncoder(&termData).Encode(cm.currentTerm); err != nil {
log.Fatal(err)
}
cm.storage.Set("currentTerm", termData.Bytes())
var votedData bytes.Buffer
if err := gob.NewEncoder(&votedData).Encode(cm.votedFor); err != nil {
log.Fatal(err)
}
cm.storage.Set("votedFor", votedData.Bytes())
var logData bytes.Buffer
if err := gob.NewEncoder(&logData).Encode(cm.log); err != nil {
log.Fatal(err)
}
cm.storage.Set("log", logData.Bytes())
}
Persistence is implemented straightforwardly: persistToStorage is invoked at every point where any of these state variables changes. That's a handful of call sites across the management methods. It is admittedly not efficient — especially saving the whole log on each update — but it is simple and correct. Real deployments rely on log compaction (section 7 of the paper); adding it here is left as an exercise.
Resilience and Unreliable RPCs
With persistence in place, a majority of servers surviving crashes keeps the cluster available, possibly after a brief leader election if a crashed peer was the leader. A cluster of 2N+1 servers tolerates N failures. The tests in this part include many crash scenarios; running a few and observing the recovery behavior is instructive.
The tests also exercise unreliable network delivery. The RPCProxy type in server.go already simulates realistic network delays of 1–5 ms for RPCs. With the RAFT_UNRELIABLE_RPC environment variable enabled, some RPCs are delayed by as much as 75 ms or dropped entirely. Re-running all tests with this setting demonstrates how Raft behaves under network glitches. An optional exercise is to extend RPCProxy to also delay RPC replies, not just requests.
Removing Latency from AppendEntries
In Part 2, leaders sent AppendEntries (AEs) only via a periodic timer firing every 50 ms. This introduced unnecessary latency, as the figure shows:
Upon receiving a new command via Submit, the leader waits until the next timer boundary to send the updated log (step 2). After the follower acknowledges (step 3) — and the leader advances its commit index — the leader waits for yet another tick to transmit the new leaderCommit (step 4). This doubles the round-trip time imposed by the periodic heartbeat pattern.
The optimized behavior is shown here:
The fix is to trigger an AE send immediately when something happens that requires followers to be updated, rather than only on a tick. The startLeader loop now selects between two events:
- A send on
cm.triggerAEChan - A 50 ms timer for heartbeats, which resets whenever the channel fires
The method that broadcasts AEs is renamed from leaderSendHeartbeats to leaderSendAEs for accuracy. The loop logic is:
func (cm *ConsensusModule) startLeader() {
cm.state = Leader
for _, peerId := range cm.peerIds {
cm.nextIndex[peerId] = len(cm.log)
cm.matchIndex[peerId] = -1
}
cm.dlog("becomes Leader; term=%d, nextIndex=%v, matchIndex=%v; log=%v", cm.currentTerm, cm.nextIndex, cm.matchIndex, cm.log)
// This goroutine runs in the background and sends AEs to peers:
// * Whenever something is sent on triggerAEChan
// * ... Or every 50 ms, if no events occur on triggerAEChan
go func(heartbeatTimeout time.Duration) {
// Immediately send AEs to peers.
cm.leaderSendAEs()
t := time.NewTimer(heartbeatTimeout)
defer t.Stop()
for {
doSend := false
select {
case <-t.C:
doSend = true
// Reset timer to fire again after heartbeatTimeout.
t.Stop()
t.Reset(heartbeatTimeout)
case _, ok := <-cm.triggerAEChan:
if ok {
doSend = true
} else {
return
}
// Reset timer for heartbeatTimeout.
if !t.Stop() {
<-t.C
}
t.Reset(heartbeatTimeout)
}
if doSend {
cm.mu.Lock()
if cm.state != Leader {
cm.mu.Unlock()
return
}
cm.mu.Unlock()
cm.leaderSendAEs()
}
}
}(50 * time.Millisecond)
}
Submit is one source of trigger signals, together with a newly added persistence call for the appended log entry:
func (cm *ConsensusModule) Submit(command any) int {
cm.mu.Lock()
cm.dlog("Submit received by %v: %v", cm.state, command)
if cm.state == Leader {
submitIndex := len(cm.log)
cm.log = append(cm.log, LogEntry{Command: command, Term: cm.currentTerm})
cm.persistToStorage()
cm.dlog("... log=%v", cm.log)
cm.mu.Unlock()
cm.triggerAEChan <- struct{}{}
return submitIndex
}
cm.mu.Unlock()
return -1
}
Notable changes here: an empty struct is sent on triggerAEChan when a command is appended, lock handling is reordered to avoid a deadlock while sending on that channel, and Submit now returns the log index rather than a boolean (useful in Part 4).
The second trigger point is inside the AE-reply handling code where the leader advances its commit index:
if cm.commitIndex != savedCommitIndex {
cm.dlog("leader sets commitIndex := %d", cm.commitIndex)
// Commit index changed: the leader considers new entries to be
// committed. Send new entries on the commit channel to this
// leader's clients, and notify followers by sending them AEs.
cm.newCommitReadyChan <- struct{}{}
cm.triggerAEChan <- struct{}{}
}
Together these two triggers ensure that leaders notify followers about uncommitted log entries and committed indexes with minimal delay.
Batching Submissions
One might worry that triggering AE sends on every Submit floods the network when many commands arrive at once. In practice, this is safe because Raft RPCs are idempotent — receiving duplicate AEs with the same information causes no harm. If network traffic is a concern, however, batching is straightforward to add: allow Submit to accept a slice of commands, and change very little inside the Raft implementation. This is left as a suggested exercise.
Backtracking More Efficiently on AppendEntries Rejections
A common optimization in Raft implementations addresses the cost of bringing a severely out-of-date follower up to speed. By default, nextIndex begins at the end of the leader's log and decrements by one per rejected AppendEntries RPC. When a follower lags by thousands of entries, this linear backtracking can take a long time, as each round-trip only advances a single log position.
Section 5.3 of the Raft paper mentions this problem but provides little implementation detail. To make the optimization concrete, the AppendEntries reply gets extended with two additional fields:
type AppendEntriesReply struct {
Term int
Success bool
// Faster conflict resolution optimization (described near the end of section
// 5.3 in the paper.)
ConflictIndex int
ConflictTerm int
}
Two code paths change to support this:
- The
AppendEntriesRPC handler on the follower now populatesConflictIndexandConflictTermwhen it rejects an AE request. leaderSendAEs, which processes AE replies, uses those two fields to jumpnextIndexbackward more aggressively instead of stepping one entry at a time.
The Raft paper itself cautions:
In practice, we doubt this optimization is necessary, since failures happen infrequently and it is unlikely that there will be many inconsistent entries.
That skepticism matches experience. Reproducing this scenario for testing requires a highly contrived setup. The realistic frequency of such divergence is vanishingly small; the occasional savings of a few hundred milliseconds does not justify the added complexity in the common path. This optimization is a useful illustration of how the Raft algorithm can be adapted for uncommon edge cases, but it is not central to correctness or typical performance.
Raft deliberately optimizes the common case: stable leaders, timely heartbeats, and fast log replication. Optimizations like immediate AE delivery directly improve that primary path. Conflict indices, by contrast, only pay off in rare failure cascades that occupy a negligible fraction of a cluster's lifetime.
Wrapping Up the Core Implementation
With persistence and these optimizations in place, the core Raft implementation is complete. A follow-up part builds a realistic key/value database on top of this foundation, demonstrating how the consensus module integrates with an application layer.
For production-grade implementations in Go, two solid references exist with battle-tested code:
etcd/raftimplements the Raft portion of the etcd distributed key-value store.hashicorp/raftis a standalone Raft module designed to be plugged into different client systems.
These libraries cover advanced features from the Raft paper that a minimal implementation does not, including:
- Cluster membership changes (Section 6) for replacing permanently offline servers without downtime.
- Log compaction (Section 7) to checkpoint and truncate logs, which otherwise grow unboundedly in real deployments.
