Replication copies bytes; consensus decides which command is entry N, once, for every replica. Deterministic state machines that apply the same log in the same order end in the same state — that is the whole product. In Raft an entry is committed when the leader that created it has it on stable storage of a majority; only then is it applied and acknowledged. Every message carries a term (an epoch): a server that sees a higher term steps down, and a request with a stale term is rejected. Safety — no two replicas apply different commands at the same index — must never depend on timing; only liveness does.
// Default Get is linearizable: it goes through Raft and reflects every acknowledged write.
cur, err := cli.Get(ctx, "config/limits")
if err != nil {
return err
}
// Serializable: answered by the local member, may lag the quorum.
if _, err := cli.Get(ctx, "config/limits", clientv3.WithSerializable()); err != nil {
return err
}
rev := int64(0) // a missing key compares as ModRevision 0
if len(cur.Kvs) == 1 {
rev = cur.Kvs[0].ModRevision
}
// Compare-and-swap on the revision we read; the replicated log orders the contenders.
txn, err := cli.Txn(ctx).
If(clientv3.Compare(clientv3.ModRevision("config/limits"), "=", rev)).
Then(clientv3.OpPut("config/limits", next)).
Commit()
if err != nil {
return err // outcome unknown: the write may have committed; re-read before retrying
}
if !txn.Succeeded {
return errLostRace // someone committed in between: re-read, recompute, retry
}Where the theorem ends. FLP proves that no deterministic protocol guarantees agreement and termination in a fully asynchronous system if even one process may crash. Paxos and Raft answer by never giving up safety and buying termination with timing assumptions: Raft keeps a stable leader while broadcastTime ≪ electionTimeout ≪ MTBF, and randomized election timeouts make split votes unlikely, not impossible. The engineering consequences: a majority of 2f+1 servers tolerates f failures, so a fourth server adds cost and no tolerance; during an election or a partition the minority side must refuse writes — the C in 64; and every persisted term, vote and log entry is fsynced before the reply, or a restart can vote twice.
The three places implementations get it wrong. Reads: a leader answering from local state may already be deposed. Raft’s linearizable read commits a no-op at the start of the term and confirms leadership with a majority heartbeat before answering; a leader lease skips the round trip but makes safety depend on bounded clock drift (67) — the one place the timing rule is knowingly bent. Membership: changing the voter set must keep old and new majorities overlapping (joint consensus, or one server at a time); the single-server scheme published in 2014 had a split-brain bug fixed by one rule — a leader appends a configuration entry only after committing an entry from its own term. Clients: a retry after a timeout can apply a command twice; linearizable semantics need client ids and sequence numbers deduplicated inside the state machine (22).
Consensus orders commands inside the group. A lock or leadership taken from it protects an external resource only with a fencing token checked by that resource (66); an etcd revision like the one above can serve as that token. Use an existing implementation (etcd, ZooKeeper, a database’s built-in quorum) and test it under faults rather than trusting the paper’s proof for your code (98, 99).