Failure Models & CAP
Reason like a staff engineer: why a timeout tells you almost nothing, CAP as a per-partition choice rather than pick-2-of-3, and real databases placed on the PACELC spectrum with their latency tax named.
Partial Failure & the Fallacies of Distributed Computing
A timeout is fundamentally ambiguous across four physical outcomes, so every call must be designed for retries, reordering, duplication, and stale reads via idempotency.
CAP Theorem (Correct Framing)
CAP forces a choice only during a partition: linearizability or availability. P is non-negotiable, CA is not real, and production systems tune the choice per operation.
PACELC & the Steady-State Tradeoff
The else-case tax: linearizable reads and writes cost leader or quorum round trips on every request, so place stores on PA/EL vs PC/EC and tie the choice to a latency SLO.
Consistency & Time
Place any system precisely on the consistency spectrum with its coordination cost, fix staleness bugs with the four session guarantees, order events without a shared clock, and treat clock drift as a correctness input.
Consistency Models Spectrum
Linearizable, sequential, causal, eventual: name the exact model and its coordination cost, and always pick the weakest model that is still correct for the data.
Client-Centric Session Guarantees
Fix per-client staleness bugs with the four Bayou session guarantees, implemented via sticky routing or version tokens, with tokens required for cross-device correctness.
Logical Time: Lamport & Vector Clocks
Lamport clocks give a causality-respecting total order but cannot detect concurrency; vector clocks detect it and surface siblings, at O(N) size with a GC problem.
Physical Time, Clock Uncertainty, HLC & TrueTime
Wall-clock LWW silently drops writes under NTP skew; HLC gives causal, monotonic timestamps on commodity hardware, TrueTime's bounded interval plus commit-wait buys external consistency.
Consensus & Coordination
Replicating a service correctly reduces to agreeing on an ordered log; reason through Raft across a leader crash, and pick concrete N/R/W quorum settings while naming exactly what consistency you get.
State-Machine Replication & Total-Order Broadcast
Identical replicas come from deterministic commands applied in one agreed order; that order is total-order broadcast, equivalent to consensus, with snapshots bounding the log.
Consensus in Depth: Raft (and the Paxos Family)
Randomized-timeout election, majority-quorum commit, and four safety properties make committed entries immortal; minority partitions stall safely and clusters should be odd-sized.
Quorums & Dynamo-Style Tunable Consistency
R+W>N forces read/write overlap (quorum consistency, NOT linearizability); latency tracks the slowest quorum member, and sloppy quorums buy availability during partitions.
Distributed Transactions
Atomicity across independently-owned services: why 2PC blocks at scale, how sagas trade isolation for progress, how the outbox kills the dual-write problem, and why exactly-once is an application property.
Distributed Transactions: 2PC / 3PC & Their Limits
2PC gives atomicity via prepare-then-commit, but a coordinator crash after the vote blocks participants holding locks; harden the coordinator with consensus or use sagas.
Sagas: Orchestration vs Choreography & Compensation
Chain local transactions with compensating undos: atomicity of outcome without isolation, contained by semantic locks, with idempotent compensations backed by retries and a DLQ.
Transactional Messaging: Outbox, Inbox & CDC
The dual write either loses or fabricates events; write the event to an outbox in the same local transaction, relay it at-least-once, and dedupe with a consumer inbox.
Delivery Semantics, Idempotency & Exactly-Once Reality
Exactly-once delivery is impossible; build exactly-once effect from at-least-once plus idempotency keys stored atomically with the side effect, with fencing tokens for stale actors.
Membership & Failure Handling
Converge replicas without coordination via CRDTs and anti-entropy, detect crashes without falsely evicting slow nodes, prevent split-brain with leases and fencing tokens, and know when BFT is justified.
CRDTs, Strong Eventual Consistency & Anti-Entropy
Commutative, associative, idempotent merges give deterministic convergence, at the cost of tombstones and no global invariants, and only with anti-entropy delivering the missed writes.
Failure Detection: Heartbeats, Phi-Accrual & SWIM
Dead and slow are indistinguishable, so use phi-accrual's adaptive suspicion, a refutation window, and SWIM's O(1) probes with gossip instead of one fixed timeout.
Leader Election, Leases, Fencing & Split-Brain
A GC pause can create two leaders despite a valid lease, so enforce monotonic fencing tokens at the storage layer; on a 3-2 split the majority leads and the minority is fenced.
Byzantine Fault Tolerance & BFT Consensus
Byzantine nodes lie and equivocate, forcing 3f+1 nodes and heavier messaging (PBFT, linear HotStuff); use BFT only across real trust boundaries, Raft plus checksums/TLS inside one.