Distributed consensus, replication, and fault tolerance / personal lessonsLesson 13 of 17
Lesson browserDistributed consensus, replication, and fault tolerance lessonsPrevious: Retries and the boundary of exactly onceNext: Consensus is not a distributed transaction
Distributed systems · Lesson 13

Recovery, snapshots, and membership changes

Preserve protocol memory while compacting logs and replacing machines.

By the end: Explain why recovery and reconfiguration are protocol operations, not file-management shortcuts.

Recover a consistent state, not just some files

A restarting server must reconstruct both application state and protocol obligations. In Raft, current term, vote, and log entries have durable requirements. Application snapshots must correspond to a known applied committed prefix. A snapshot of half an operation is not a valid state-machine checkpoint even if its files are individually intact.

A snapshot can replace a long prefix with the state obtained by applying it. Keep the last included log index and term so future prefix checks know where the remaining suffix attaches. Include the relevant membership and application metadata, such as deduplication results. Preserve a consistent boundary between the snapshot and log; do not delete the only recoverable prefix before its replacement is safely stored. Snapshot transfer also needs completeness and integrity checks before installation.

A snapshot replaces an exact execution prefix

A log and a snapshot describe the same state in different ways. The log records the commands that produced it. A snapshot records their accumulated result at one boundary. Suppose committed entries 1 through 40 have been applied, but entries 41 through 45 are merely stored. A snapshot at 40 may replace the first forty commands. It must not include an effect from 44 while omitting an effect from 42. That mixture would not describe any prefix of the agreed execution, so replaying the remaining suffix could repeat some effects and omit others.

Consider a separate paper example with two account balances and an operation-result table in one replicated state machine. At index 39, Ada has 70 and Bo has 30. Entry 40 transfers 10 from Ada to Bo and records result K as successful. After application, the balances are 60 and 40. A snapshot taken across this transition must contain either the complete state at 39 or the complete state at 40. Copying Ada's balance before application and Bo's balance afterward would manufacture a total of 110. Copying both new balances but omitting K would preserve the total while allowing a retry to repeat the transfer.

A running service therefore needs a consistent snapshot mechanism. One possible approach briefly stops application while copying the state. Another captures a stable database version while application continues against a later version. These are implementation choices, not features of this lesson's table. The required property is the same in both cases. The bytes must represent all application transitions through a chosen applied index and none afterward. Stored but unapplied log entries are not evidence that their effects belong in that image.

The last included index tells recovery where replay resumes. The last included entry term identifies the log boundary for Raft's matching rules. If a snapshot ends at index 40 with entry term 6, that does not mean the recovering server's current election term is 6. It might already have advanced to election term 9 and promised a vote there. Snapshot metadata about a past entry cannot erase that newer protocol obligation. Application state, log boundary metadata, and persistent election state have distinct meanings even when the storage implementation packages them together.

Track a hypothetical restart by writing three lines on paper. The first names the snapshot's applied boundary. The second lists the retained suffix. The third records the current election term and vote. With a snapshot through 40 and a retained suffix starting at 41, recovery reconstructs state at 40 before any later committed entries are applied. An unapplied entry is not automatically discarded, nor is it automatically executed. The replication protocol still determines which continuation is authoritative. This reconstruction assumes the snapshot and required persistent records survive the stated process crash. It does not establish safety after arbitrary disk corruption or loss of every durable copy.

Change who counts without creating two authorities

Suppose an old configuration is A,B,C and the new one is C,D,E. An old majority A+B and a new majority D+E do not intersect. If each configuration can independently act as the whole cluster during a transition, two separate histories can be approved. Sharing one name C in the membership lists does not mean every pair of quorums shares C.

Raft’s joint-consensus transition requires separate majorities of both old and new configurations during the joint phase. A,B,D,E would satisfy it; A,B alone would not, nor would D,E alone. Once the joint configuration is committed, the protocol can move to the new configuration under its transition rules. Configuration entries themselves are replicated decisions. This is a conceptual account of the joint approach, not an instruction to hand-edit server configurations; implementations may expose different safe reconfiguration procedures.

Persist the replacement before discarding its history

Compaction changes the recovery material while the service is running. That creates a second atomicity problem beyond capturing a consistent application state. Suppose the old snapshot ends at 20 and the log covers 21 through 60. The service prepares a new snapshot through 50. If it deletes entries 21 through 50 before the replacement snapshot is durably available, a crash can leave the old snapshot and only entries 51 onward. There is then no way to reconstruct the transitions between 20 and 50 from those files. The lost material may include committed commands.

A safe design keeps the old recovery path until it has installed the new one under the storage system's durability rules. Conceptually, it writes a complete candidate snapshot, makes its contents durable, publishes metadata selecting that snapshot, and only then reclaims obsolete log data. Actual file-system and database operations differ, so this ordering is a requirement rather than a portable recipe involving one particular file operation. Recovery must be able to distinguish a completed installation from abandoned temporary bytes after a crash. A checksum can detect some corruption, but cannot prove that a snapshot contains the correct application prefix.

Snapshot transfer has the same distinction between arrival and installation. Imagine a follower receiving ten chunks. Nine complete chunks do not form a snapshot through index 50. The follower needs a way to identify the intended snapshot, assemble its pieces, check completeness, and install the result consistently. It must not advertise successful recovery merely because the final network request arrived. Nor may an old transfer overwrite a newer installed state without the protocol's boundary checks. Interrupted transfers and repeated chunks are expected recovery cases, not reasons to reinterpret partial state as complete.

Membership changes add a separate question about which servers can authorize the continuation. In the old configuration A,B,C, a majority needs two responses. In the new configuration C,D,E, a majority also needs two. The union contains five identities, but requiring any three of those five is not the joint rule. For example, A,B,D are three union members yet include only D from the new configuration. If that set alone could authorize a joint decision, the new configuration's required evidence would be missing. C's presence in both lists does not turn every three-member set into two majorities.

Before reading the table below, mark each responding identity once in the old list and once in the new list where applicable. Count C in both lists because it belongs to both, not because it sent two independent votes. The joint condition requires at least two old members and at least two new members. This arithmetic explains the table's acceptance condition. It does not specify the complete log rules for entering and leaving the joint phase. Those replicated configuration transitions must follow the chosen implementation's protocol, including its rules about when each configuration takes effect.

Work through the model

Shared membership is not shared authority

Separate majorities: A B versus D EA✓B✓CD✓E✓Old: A B CNew: C D EJoint response set: B C DAB✓C✓D✓EOld: A B CNew: C D E
Checkmarks identify the named responders. In the upper row A+B can form an old majority and D+E a new majority without any common responder. If all four actually respond to one joint decision, that combined set can satisfy both requirements. The lower row shows a smaller valid joint set: B+C is an old majority, while C+D is a new majority. C's single response counts once in each configuration, not twice within either one.
StateOld A B C votesNew C D E votesJoint decision?
A+B only2 of 30 of 3No
D+E only0 of 32 of 3No
A+B+D+E2 of 32 of 3Yes
B+C+D2 of 32 of 3Yes
Joint consensus requires two majorities, not a majority of the union’s size.

Separate catching up from granting a vote

The displayed membership table compares response sets, not a running reconfiguration engine. Each row answers whether those identities meet the stated joint quorum condition. In A+B+D+E, the old majority is A+B and the new majority is D+E. Although these two subsets do not intersect each other, together they satisfy the transition requirement. In B+C+D, the old majority is B+C and the new majority is C+D. Joint decisions cannot be justified by either configuration acting alone. The safety argument uses the transition protocol and the intersections required across successive decisions, not a claim that every old majority intersects every new majority.

A separate catch-up example shows why possession of history and voting membership should not be confused. Begin with three voters A,B,C. The service can form a majority with two. Add an empty D immediately as a fourth voter and the threshold rises to three. If C is unavailable and D cannot yet supply the required protocol responses, A+B no longer suffice. Adding a machine intended to improve resilience has reduced the currently usable voting margin. This is a threshold effect, not evidence that larger clusters always perform worse.

A learner receives state without yet counting toward voting majorities. It can install a snapshot and then receive the later suffix before promotion. Promotion still needs a safe configuration change. Catch-up does not authorize an operator to edit the voter list independently on each machine. Also, a durable response from D must mean what the protocol expects. Counting a response from a process that stores entries only in volatile memory would make a threshold look satisfied while removing the crash-survival property the count was supposed to support.

Now give D a snapshot through 100 while the leader has applied through 130. The initial lag is thirty positions. While D receives those entries, the leader may append ten more. D must reduce the gap rather than merely receive a snapshot once. Catch-up depends on transfer capacity, ongoing write volume, and stable enough communication. No fixed number of seconds follows from the index difference alone. The lesson's static table exposes neither bandwidth nor application time, so it cannot predict a promotion delay or prove that a particular learner is ready.

Identity replacement also needs an explicit state transition. Suppose B voted in term 12, then its disk is lost. Starting an empty process with B's old name does not restore that vote or any accepted log prefix. Other servers may still rely on B's earlier promises. The safe replacement procedure must account for the lost memory and configured membership, rather than treating the name as proof of continuity. This differs from restarting the same server with intact durable state. For paper analysis, label these as different failures even if an operations dashboard calls both events a restart.

Catch up, replace, and recover deliberately

A new machine can first join as a nonvoting learner so it receives history before its vote changes availability requirements. Adding an empty voter immediately can raise the required majority while adding no ready responder. Identity also matters: replacing a failed server by reusing an identity with empty state must follow the product’s recovery protocol, rather than pretending the new process remembers the old promises.

Losing a majority can stop progress while leaving the committed history intact on surviving disks. Forcing a new cluster from one survivor may discard acknowledged data if that survivor lacks it or allow an old cluster to return as a competing authority. Disaster recovery requires explicit fencing of old participants and a documented data-loss boundary. A backup restore is not an ordinary leader election. Practice restoration and compare the recovered state against the service’s actual recovery-point and recovery-time objectives.

A restore creates a new recovery obligation

Losing quorum reachability and losing committed data are different incidents. Suppose A+B committed entry 80 in a three-voter cluster. A and B are unreachable, while C has entries only through 75. C's inability to make progress does not prove entries 76 through 80 have disappeared. The disks at A and B may still contain them. Conversely, letting C proceed alone cannot prove those entries will survive. Its local state lacks the evidence required to reconstruct them. Restoring service from C therefore changes the recovery claim if no later history can be obtained.

Disaster recovery must also prevent the former authority from returning independently. If an operator creates a replacement cluster from C while A+B continue to run elsewhere, two groups can accept incompatible commands under different memberships. A forced recovery needs an operational method to exclude the old authority and redirect clients. That may involve revoking access to shared resources, disabling old endpoints, or other product-specific fencing. Changing a local label to new cluster does not itself prevent old processes from serving requests or producing external effects. The exact enforcement point must be named in the recovery plan.

Recovery-point and recovery-time objectives answer different questions. In a constructed example, a backup contains state through a checkpoint made at noon, and the service fails at 12:08. A noon recovery point describes what history is retained, not proof that exactly eight minutes of successful work are lost. There may have been no writes, or additional durable logs may be recoverable. A twenty-minute recovery-time objective concerns how soon usable service should return. Meeting that duration while silently violating the promised recovery point is not a successful restoration under both objectives.

A useful restoration rehearsal follows data with several kinds of obligations. Include a command whose reply was lost, a retained deduplication result, a snapshot boundary, and a later committed suffix. Reconstruct the application state and ask which operation identities remain recognized. Then distinguish an internal replay from a newly delivered external event. An outbox record restored without a later delivery acknowledgment can be sent again, so the receiver's durable deduplication contract still matters. Recovery of database rows does not make the external system forget effects it already completed.

Finally, state the limit of the rehearsal. Matching a restored balance checks one projection of state. It does not establish that votes, memberships, request results, and external-delivery obligations are all correct. A stronger check compares the exact recoverable boundary and the semantic records needed by the service's contract. Even that check is evidence for the tested failure and restore procedure, not a proof over arbitrary simultaneous disk loss or every possible implementation bug. Ordinary crash recovery preserves existing obligations. A disaster procedure must identify any obligations it cannot preserve before presenting restored service as equivalent to the original cluster.

Exercises

A snapshot loses request identities

A snapshot preserves balances but discards all deduplication keys for retained operations. What client guarantee can fail after recovery?

Hint

Replay a previously successful operation whose reply was lost.

Worked answer
Recovered fieldPresent?Effect
BalanceYesOriginal debit retained
Request K resultNoRetry K may debit again
Recovery must preserve semantic state, not just visible business rows.

A late retry can look new and repeat the balance change. The snapshot must preserve the deduplication state required by the advertised retention contract, or the API must explicitly end that contract through a safe session/epoch policy. Application values alone are not the complete state.

One shared machine in two configurations

Old members are A,B,C; new members are C,D,E. Does sharing C make an immediate switch safe?

Hint

Find an old majority and a new majority that omit C.

Worked answer
QuorumMembersIncludes C?
Old majorityA BNo
New majorityD ENo
Membership overlap is weaker than quorum intersection.

No. A+B and D+E are disjoint majorities. A controlled transition must prevent them from independently authorizing conflicting histories. In the joint phase, neither pair alone is enough; both old and new majority conditions must hold.

Sources and further reading

The explanations and worked answers above stand alone. These references provide the original algorithms, assumptions, and deeper proofs; they are not required downloads.

Lesson browserDistributed consensus, replication, and fault tolerance lessonsPrevious: Retries and the boundary of exactly onceNext: Consensus is not a distributed transaction