Anatomy of the implementation

To study Raft in isolation from real-world service concerns, the code in this series separates the consensus logic into its own type. A ConsensusModule (in raft.go) contains everything that makes the algorithm work, while a thin Server scaffold (in server.go) handles networking:

Architecture of a consensus module embedded into a server

The consensus module is deliberately oblivious to transport details. Its only networking-adjacent fields point back to the owning server and list the cluster's peer IDs:

// id is the server ID of this CM.
id int

// peerIds lists the IDs of our peers in the cluster.
peerIds []int

// server is the server containing this CM. It's used to issue RPC calls
// to peers.
server *Server

Each replica refers to the other members of the cluster as its peers, identified by unique numeric IDs. The server field lets the module dispatch RPCs to those peers. Any method on ConsensusModule that carries out the algorithm is self-contained — mapping the paper's logic onto code does not require understanding the server scaffold underneath.

States and terms

Internally, a Raft replica is a state machine with three states:

Raft high level state machine (Figure 4 from Raft paper)

Following the terminology from part 0, this can read as an odd statement — Raft is the algorithm that lets you build replicated state machines, and yet it contains a small state machine of its own. The context usually makes it clear which "state" is meant. Steady-state operation sees a single leader with the rest as followers. Failure handling is where the algorithm spends most of its effort: any follower can time out and transition to candidate, triggering a new election.

Elections run in terms. A term spans the period during which a particular leader is in charge, and the algorithm enforces that a given term can have only one leader. Unlike real-world elections, Raft's are cooperative: candidates compete to win, but they share the higher-level goal of electing some viable server in each term.

Election timing

Every follower runs an election timer. Each time it hears from a valid leader (via a heartbeat or equivalent), the timer resets. If it expires, the follower assumes the leader is gone and transitions to candidate.

Randomization is what keeps the cluster from turning all followers into candidates simultaneously. Even when the vote splits and no candidate reaches a majority in a term, a new election with a higher term will eventually break the tie. Repeated deadlocks are theoretically possible but become probabilistically negligible after each round.

A partitioned follower will also time out and start elections. If it is the one cut off, its vote requests simply go unanswered and it continues cycling through candidate terms until connectivity returns. Raft does not attempt to distinguish "who is partitioned" — the outcome emerges from how the protocol behaves.

Two RPCs, many roles

Raft defines only two types of inter-peer RPC:

  • RequestVotes (RV) — candidates send this to peers when asking for votes. The reply states whether the vote was granted.
  • AppendEntries (AE) — leaders send this to replicate log entries, and also as a heartbeat when there is no new data to send.

Followers never initiate RPCs. Their job is entirely reactive: answer RVs and AEs from peers, and wait for the election timer. When it fires, they turn into candidates and begin sending RVs themselves.

Election timer in code

The election timer runs in a dedicated goroutine:

func (cm *ConsensusModule) runElectionTimer() {
  timeoutDuration := cm.electionTimeout()
  cm.mu.Lock()
  termStarted := cm.currentTerm
  cm.mu.Unlock()
  cm.dlog("election timer started (%v), term=%d", timeoutDuration, termStarted)

  // This loops until either:
  // - we discover the election timer is no longer needed, or
  // - the election timer expires and this CM becomes a candidate
  // In a follower, this typically keeps running in the background for the
  // duration of the CM's lifetime.
  ticker := time.NewTicker(10 * time.Millisecond)
  defer ticker.Stop()
  for {
    <-ticker.C

    cm.mu.Lock()
    if cm.state != Candidate && cm.state != Follower {
      cm.dlog("in election timer state=%s, bailing out", cm.state)
      cm.mu.Unlock()
      return
    }

    if termStarted != cm.currentTerm {
      cm.dlog("in election timer term changed from %d to %d, bailing out", termStarted, cm.currentTerm)
      cm.mu.Unlock()
      return
    }

    // Start an election if we haven't heard from a leader or haven't voted for
    // someone for the duration of the timeout.
    if elapsed := time.Since(cm.electionResetEvent); elapsed >= timeoutDuration {
      cm.startElection()
      cm.mu.Unlock()
      return
    }
    cm.mu.Unlock()
  }
}

Each iteration starts by picking a random election timeout in the 150–300 ms range recommended by the paper. The function runs a 10 ms ticker, polling until that timeout expires. This polling approach adds a little latency but keeps the code straightforward to read and debug. Every iteration starts with a lock of the ConsensusModule, since the implementation is mostly synchronous (a natural fit for Go) but still must guard shared state from concurrent RPC handlers.

The loop exits early if the replica finds itself in a state or term it no longer expects — for instance, another server won the election while the timer was running. Two events reset the timer in practice: receipt of a valid heartbeat from the current leader, and casting a vote for another candidate. Both are handled elsewhere in the module.

Starting an election

When a follower's timer expires, a candidate needs three things: a new term, votes gathered from peers, and a decision about whether the vote count is sufficient. One function covers all of that:

func (cm *ConsensusModule) startElection() {
  cm.state = Candidate
  cm.currentTerm += 1
  savedCurrentTerm := cm.currentTerm
  cm.electionResetEvent = time.Now()
  cm.votedFor = cm.id
  cm.dlog("becomes Candidate (currentTerm=%d); log=%v", savedCurrentTerm, cm.log)

  votesReceived := 1

  // Send RequestVote RPCs to all other servers concurrently.
  for _, peerId := range cm.peerIds {
    go func(peerId int) {
      args := RequestVoteArgs{
        Term:        savedCurrentTerm,
        CandidateId: cm.id,
      }
      var reply RequestVoteReply

      cm.dlog("sending RequestVote to %d: %+v", peerId, args)
      if err := cm.server.Call(peerId, "ConsensusModule.RequestVote", args, &reply); err == nil {
        cm.mu.Lock()
        defer cm.mu.Unlock()
        cm.dlog("received RequestVoteReply %+v", reply)

        if cm.state != Candidate {
          cm.dlog("while waiting for reply, state = %v", cm.state)
          return
        }

        if reply.Term > savedCurrentTerm {
          cm.dlog("term out of date in RequestVoteReply")
          cm.becomeFollower(reply.Term)
          return
        } else if reply.Term == savedCurrentTerm {
          if reply.VoteGranted {
            votesReceived++
            if votesReceived*2 > len(cm.peerIds)+1 {
              // Won the election!
              cm.dlog("wins election with %d votes", votesReceived)
              cm.startLeader()
              return
            }
          }
        }
      }
    }(peerId)
  }

  // Run another election timer, in case this election is not successful.
  go cm.runElectionTimer()
}

The candidate votes for itself immediately — votesReceived is set to 1 and votedFor to its own ID — then fires a goroutine for each peer. RPC calls are synchronous, so parallel goroutines are the only practical way to reach everyone without waiting on one peer at a time. Here is the call pattern:

cm.server.Call(peer, "ConsensusModule.RequestVote", args, &reply)

That snippet routes through the embedded Server, invoking the peer's RequestVote method. Every reply must be evaluated against the current state:

  • If this replica is no longer a candidate — for example, it won the election based on votes from the other goroutines — the result is ignored.
  • If the reply carries a higher term than the one in which the request was sent, this replica immediately reverts to follower. This handles the case where another candidate won while votes were in flight.
  • Otherwise, a granted vote is tallied. Reaching a majority (the self-vote included) transitions the replica to leader.

startElection returns as soon as it has launched those goroutines. Its final line fires a fresh election timer, which keeps elections rolling indefinitely as long as no candidate can secure a majority. If the replica did become a leader meanwhile, that concurrent timer will quietly exit on its next state check.

Taking leadership

Winning a vote count hands control to startLeader:

func (cm *ConsensusModule) startLeader() {
  cm.state = Leader
  cm.dlog("becomes Leader; term=%d, log=%v", cm.currentTerm, cm.log)

  go func() {
    ticker := time.NewTicker(50 * time.Millisecond)
    defer ticker.Stop()

    // Send periodic heartbeats, as long as still leader.
    for {
      cm.leaderSendHeartbeats()
      <-ticker.C

      cm.mu.Lock()
      if cm.state != Leader {
        cm.mu.Unlock()
        return
      }
      cm.mu.Unlock()
    }
  }()
}

All it does is start a heartbeat ticker that calls leaderSendHeartbeats every 50 ms while the server remains the leader. That routine is the mirror image of startElection — a goroutine per peer, this time sending a log-less AppendEntries:

func (cm *ConsensusModule) leaderSendHeartbeats() {
  cm.mu.Lock()
  savedCurrentTerm := cm.currentTerm
  cm.mu.Unlock()

  for _, peerId := range cm.peerIds {
    args := AppendEntriesArgs{
      Term:     savedCurrentTerm,
      LeaderId: cm.id,
    }
    go func(peerId int) {
      cm.dlog("sending AppendEntries to %v: ni=%d, args=%+v", peerId, 0, args)
      var reply AppendEntriesReply
      if err := cm.server.Call(peerId, "ConsensusModule.AppendEntries", args, &reply); err == nil {
        cm.mu.Lock()
        defer cm.mu.Unlock()
        if reply.Term > savedCurrentTerm {
          cm.dlog("term out of date in heartbeat reply")
          cm.becomeFollower(reply.Term)
          return
        }
      }
    }(peerId)
  }
}

A reply with a higher term demotes the leader back into a follower, just as it did for a candidate. The final piece is becomeFollower:

func (cm *ConsensusModule) becomeFollower(term int) {
  cm.dlog("becomes Follower with term=%d; log=%v", term, cm.log)
  cm.state = Follower
  cm.currentTerm = term
  cm.votedFor = -1
  cm.electionResetEvent = time.Now()

  go cm.runElectionTimer()
}

It resets the term, drops the state to follower, and — crucially — starts a new election timer, since every follower must have one running in the background at all times.

Handling Incoming RPCs

The active components—RPC initiation, timers, and state transitions—are only half of the picture. The other half is the server-side handlers that other peers invoke over the network. The RequestVote handler follows the expected pattern:

func (cm *ConsensusModule) RequestVote(args RequestVoteArgs, reply *RequestVoteReply) error {
  cm.mu.Lock()
  defer cm.mu.Unlock()
  if cm.state == Dead {
    return nil
  }
  cm.dlog("RequestVote: %+v [currentTerm=%d, votedFor=%d]", args, cm.currentTerm, cm.votedFor)

  if args.Term > cm.currentTerm {
    cm.dlog("... term out of date in RequestVote")
    cm.becomeFollower(args.Term)
  }

  if cm.currentTerm == args.Term &&
    (cm.votedFor == -1 || cm.votedFor == args.CandidateId) {
    reply.VoteGranted = true
    cm.votedFor = args.CandidateId
    cm.electionResetEvent = time.Now()
  } else {
    reply.VoteGranted = false
  }
  reply.Term = cm.currentTerm
  cm.dlog("... RequestVote reply: %+v", reply)
  return nil
}

The initial check for a Dead state handles orderly shutdown, which is discussed below. Otherwise, the logic mirrors the election rules: if the caller's term is stale, we step down to follower and reset our state. If the term matches ours and we haven't already voted for another candidate in this term, the vote is granted. Votes are never granted to callers from older terms.

The AppendEntries handler is similarly structured:

func (cm *ConsensusModule) AppendEntries(args AppendEntriesArgs, reply *AppendEntriesReply) error {
  cm.mu.Lock()
  defer cm.mu.Unlock()
  if cm.state == Dead {
    return nil
  }
  cm.dlog("AppendEntries: %+v", args)

  if args.Term > cm.currentTerm {
    cm.dlog("... term out of date in AppendEntries")
    cm.becomeFollower(args.Term)
  }

  reply.Success = false
  if args.Term == cm.currentTerm {
    if cm.state != Follower {
      cm.becomeFollower(args.Term)
    }
    cm.electionResetEvent = time.Now()
    reply.Success = true
  }

  reply.Term = cm.currentTerm
  cm.dlog("AppendEntries reply: %+v", *reply)
  return nil
}

The condition that occasionally trips people up is this one:

if cm.state != Follower {
  cm.becomeFollower(args.Term)
}

The question is: why would an existing leader step down to become a follower of another leader? The answer hinges on Raft's single-leader invariant. If you trace through the RequestVote handler and the vote-sending logic in startElection, you'll see that two leaders with the same term cannot coexist. This branch matters primarily for candidates that learn, via an AppendEntries from another peer, that they lost the election for their current term.

Goroutine Lifecycle and State Summary

Each consensus module state runs a characteristic set of goroutines:

  • Follower: Every call to becomeFollower spawns a new runElectionTimer goroutine. Brief overlaps are harmless. If a follower receives an RPC from a higher term, becomeFollower runs again and starts a fresh timer; the old one exits the moment it detects its term is outdated.
  • Candidate: In addition to the election timer, a candidate runs one goroutine per outgoing RequestVote RPC. These RPC goroutines can take a long time to return, so each one must check, upon completion, that it is still relevant—otherwise it exits quietly without taking any action.
  • Leader: No election timer runs. Instead, a single heartbeat goroutine fires every 50 ms.
  • Dead: A terminal state reached via Stop. All goroutines check for this state and exit as soon as they observe it.

With this many concurrent goroutines, leak prevention is a real concern. Several tests enable leak checking: they run a series of nontrivial election sequences, call Stop, wait for stragglers to finish, and assert that no goroutines remain unaccounted for.

The Runaway Server Scenario

A concrete example illustrates how these mechanisms interact. Consider a cluster of three servers: A, B, and C. A is the leader in term 1, sending heartbeats to B and C every 50 ms and receiving prompt responses. Both followers' electionResetEvent timestamps stay fresh.

Now a transient network fault partitions B from A and C. A's heartbeats to B fail, but the remaining two servers still form a quorum, so the cluster continues to operate. On B's side, its election timeout—say 200 ms—expires without contact from the leader. B cannot tell who is at fault, so it increments its term to 2 and starts an election. Its RequestVote RPCs to A and C are lost in the partition.

StartElection immediately launches another runElectionTimer goroutine, with a fresh random timeout between 150-300 ms. When that fires, B is still isolated, so it increments its term again and starts yet another election. This cycle repeats while the router outage lasts—by the time connectivity is restored, B's term has climbed to 8.

Reconnection happens. A, unaware of the partition's end, is still sending heartbeats every 50 ms. One of these reaches B, and B's AppendEntries handler responds with its current term, 8. A's leaderSendHeartbeats sees the higher term, updates its own term to 8, and steps down to follower. The cluster temporarily has no leader.

What happens next depends on timing. B is a candidate but may have sent its last RequestVote batch before the network healed. C is a follower whose election timer will soon expire, since A stopped sending heartbeats when it stepped down. A is now a follower in term 8 and will likewise time out into a candidate. Any of the three can win the next election.

This leader change is technically unnecessary—A never failed—and it is inefficient. That trade-off is deliberate. Raft prioritizes simple invariants over efficiency in rare corner cases; the common case, where the cluster runs undisturbed, is where performance matters, and there the protocol is efficient.

One caveat: this example assumes no new client commands were replicated while B was gone. If A and C had appended entries during the partition, their logs would be more up to date than B's, and B would be ineligible for leadership. That scenario is revisited in the next part once log replication is introduced.

Exercising the Code

The implementation is accompanied by a test suite covering specific scenarios, including the one above. Running an individual test and observing the Raft logs is highly instructive. The cm.dlog(...) calls sprinkled through the code emit diagnostic output, and the repository includes a tool that renders these logs side-by-side in an HTML view. The README explains how to use it. Adding your own dlog calls to trace code paths is encouraged.

Part 2 of this series covers the full implementation, including client command handling and log replication across the cluster.