Partial Failure & the Fallacies of Distributed Computing

No global off switch

A single program has one failure mode a programmer really cares about: it crashes, and everything stops together. A distributed system does not have an off switch. Some nodes and links fail while the rest keep running, and that is partial failure. Node A is healthy, node B is on fire, the network between them is dropping 3% of packets, and no component has a global view of any of this. Every hard result in distributed systems descends from this one fact.

The ambiguity of a timeout

When service A sends a request to service B and starts a timer, exactly one of these happened when the timer expires, and A cannot tell which:

1. The request never reached B (lost on the way out). B did nothing.
2. B received it, did the work, and the response was lost on the way back. B did the work.
3. B is just slow (GC pause, overloaded, 200ms of queueing) and the response is still coming.
4. B crashed before, during, or after the work.

A timeout is not "B failed." It is "I have no idea what B did." If A retries after case 2, B performs the side effect twice. If A gives up after case 1 when the write actually needed to happen, data is lost.

The fallacies of distributed computing

The fallacies (Deutsch, Gosling, at Sun) are the false assumptions that make people write code that ignores the four outcomes above. The load-bearing ones: the network is reliable (it is not), latency is zero (a cross-region round trip is 60 to 150ms), bandwidth is infinite (fan-out saturates links), topology is static (nodes and routes change constantly), and transport cost is zero (serialization and TLS are real CPU). Believing any of these produces a system that works in the demo and falls over in production.

Theory pins this down. In a fully asynchronous model (unbounded message delay, no clocks) you cannot even reliably tell a dead node from a slow one, which is why consensus is impossible there (FLP). Real systems assume partial synchrony: delays are usually bounded but occasionally are not, and clocks drift. That "occasionally not" is exactly where split-brain and lost writes hide.

Interview nuance: the single most common junior mistake is treating a timeout as a definite failure and re-issuing a side effect, causing a double charge or duplicate order. The senior answer: make the operation idempotent (idempotency key, dedup on the receiver) so a retry is safe regardless of which of the four outcomes actually happened. Then retries become a correctness-preserving tool instead of a bug.

The design implication: every remote interaction must assume the worst. Requests get retried (so handle duplicates), messages arrive out of order (so handle reordering), and reads can be stale. This is the baseline contract of talking over a network, not something you sprinkle in later.

   A ---- request ----> B      timeout fires. which one?
   A <--- response ---- B      (1) req lost   (2) resp lost
                               (3) B slow     (4) B dead
   A cannot distinguish them from A's side.

Recap: partial failure means parts fail independently with no global off switch, a timeout is fundamentally ambiguous (lost request, lost response, slow peer, or dead peer), the fallacies of distributed computing are the false assumptions that ignore this, and the fix is to design every call for retries, reordering, duplication, and stale reads, made safe by idempotency.