Distributed consensus, replication, and fault tolerance

Reason from failures and client guarantees to replication, consensus, recovery, and safe system design.

Prerequisite: Programming familiarity. No prior distributed-systems knowledge required. Build the reasoning needed to follow expert explanations: failure models, consistency, quorums, Raft, Paxos, safe retries, recovery, transactions, convergence, and Byzantine faults. Follow the sequence, then use the final incident to test whether you can connect the mechanisms.

Lessons

1. 1

   What a distributed system must agree on

   Separate state, commands, replicas, and the guarantees a client needs.

2. 2

   Failures, timeouts, and missing information

   Explain why silence does not distinguish a crash from a delayed message.

3. 3

   Causality and logical clocks

   Distinguish message order, wall-clock time, and concurrent events.

4. 4

   What consistency promises to a client

   Test linearizability, sequential order, causal order, and eventual convergence.

5. 5

   Replication and the meaning of success

   Trace durable appends, acknowledgments, commitment, and application.

6. 6

   Quorums and the information in an intersection

   Derive majority overlap and identify what overlap cannot prove.

7. 7

   Consensus, safety, and the limits of progress

   Understand FLP and CAP through their assumptions and indistinguishable executions.

8. 8

   Raft elections and persistent terms

   Follow votes, terms, split elections, and the log freshness restriction.

9. 9

   Raft log repair and commitment

   Explain prefix matching, leader completeness, and the current-term commitment rule.

10. 10

    Paxos and preserving an earlier decision

    Derive prepare, promises, and value adoption for one consensus slot.

11. 11

    Safe reads, leases, and fencing

    Understand stale leaders, read barriers, and protection against delayed owners.

12. 12

    Retries and the boundary of exactly once

    Separate delivery attempts, command application, and external side effects.

13. 13

    Recovery, snapshots, and membership changes

    Preserve protocol memory while compacting logs and replacing machines.

14. 14

    Consensus is not a distributed transaction

    Separate replicated decisions, atomic commit, isolation, and compensation.

15. 15

    Leaderless replication and conflict-free data

    Understand version conflicts, repair, and the limits of merging.

16. 16

    When replicas can lie

    Distinguish crash faults, equivocation, authentication, and Byzantine quorum bounds.

17. 17

    Reasoning through a distributed-system incident

    Combine guarantees, failure evidence, recovery choices, and design tradeoffs.