Streaming Durability, HA & Observability

Two promises: no acknowledged loss, and no silent failure

Operating a messaging tier comes down to two promises: an acknowledged message is never lost, and when something silently breaks you find out from a signal rather than from an angry downstream. Both have concrete, defensible settings.

No acknowledged-write loss

In Kafka terms, three settings work together. Set replication factor to 3 across three availability zones (rack-aware placement so the three replicas never share an AZ). Set the producer to acks=all, meaning the leader does not acknowledge until every in-sync replica has the record. Set min.insync.replicas=2, meaning if fewer than two replicas are in sync the producer's write is rejected rather than silently accepted with too little redundancy. The combination guarantees that any acknowledged record survives losing a full AZ, because a second copy lives in a surviving zone. Drop acks=all and a leader can ack a write that only it holds, then die, and the write is gone even though the producer got a 200.

Leader election and the unclean tradeoff

When a leader fails, a follower is promoted. If you allow unclean leader election, an out-of-sync replica can become leader, which keeps the partition available but discards records the old leader had and the new one did not: you chose availability over durability. Keep it disabled (unclean.leader.election.enable=false) when the data is money or orders; the partition goes unavailable until an in-sync replica returns, which is the correct choice when losing an acknowledged record is unacceptable.

Surviving multi-region and duplicates

For cross-region resilience you either stretch a cluster across regions (simple, but latency-bound and quorum-sensitive) or replicate asynchronously with MirrorMaker 2 to a second cluster (looser coupling, but the replica lags and failover can lose the in-flight tail). Either way, enable the idempotent producer so a producer retry across a failover does not write the record twice; combined with consumer idempotency this keeps a failover from duplicating side effects.

The signals

The primary health signal for a stream is consumer lag: the gap between the latest offset and the committed offset per partition. Rising lag means consumers are falling behind and end-to-end latency is growing, and it is the one number that turns into an SLO. Alongside it, watch under-replicated partitions (durability eroding, a replica has fallen out of sync), dead-letter-queue depth (poison messages piling up), and end-to-end latency measured with a trace that spans the async hop, not just per-service timings. Without lag and cross-hop tracing, async failures are silent: the producer got its ack, nobody is watching the consumer, and the first symptom is a stale downstream hours later.

Capacity math

Two formulas earn the offer. Partitions come from throughput: partitions ~= target throughput / per-partition throughput. If a partition sustains about 10 MB/s and you need 1M msg/s at 1 KB each (1 GB/s), that is roughly 100 partitions before you add headroom for consumer parallelism and skew (call it 150 to 200). Storage is rate x message size x retention x replication factor. At 1 GB/s, 7-day retention, replication 3: 1 GB/s x 604,800 s x 3 is on the order of 1.8 PB, so retention and replication, not raw ingest, dominate the disk bill. Network egress multiplies by fan-out: every consumer group re-reads the stream.

Interview nuance: the wrong turn is claiming durability from replication alone while leaving acks=1 or unclean election on, or having no lag metric and no trace across the async boundary so failures are invisible. Name the three durability knobs together, name lag as the SLO, and show the two capacity formulas.

Recap: prevent acknowledged loss with rack-aware RF3 plus acks=all plus min.insync.replicas=2 and clean leader election; make consumer lag the primary SLO alongside under-replicated partitions, DLQ depth, and end-to-end tracing; size partitions from throughput and storage from rate times size times retention times replication.