Design a Message Queue / Streaming Log (Kafka)

The backbone of async systems

A distributed log (Kafka, Pulsar, Kinesis) is the backbone of async systems: producers append events, consumers read them at their own pace, and the log decouples the two. The interview tests the log abstraction, delivery semantics (the famous exactly-once question), and how consumers scale.

The append-only log

The core data structure is an append-only commit log. A topic is split into partitions, and each partition is an ordered, immutable sequence of messages identified by a monotonically increasing offset. Ordering is guaranteed only within a partition, not across the topic, which is the key constraint: if you need messages for a given user in order, you must route them all to the same partition (partition by user id). This is what lets Kafka scale, because different partitions live on different brokers and are read and written in parallel.

Durability comes from replication. Each partition has a leader and followers; the leader takes writes and followers replicate. The in-sync replicas (ISR) are those caught up to the leader. A producer's acks setting controls durability: acks=1 acks after the leader writes (fast, can lose data if the leader dies before replication), acks=all acks only after all ISR replicas have the message (durable, higher latency). On leader failure a follower in the ISR is elected leader. Data is retained by time or size, or compacted (keep only the latest value per key) for changelog topics.

Delivery semantics

At-most-once means a message may be lost but never redelivered (fire and forget, no retries). At-least-once means every message is delivered but may be duplicated (retry on failure, ack after processing), which is the pragmatic default. Exactly-once is the hard one, and the crucial nuance is that exactly-once delivery over a network is impossible; what systems provide is exactly-once processing.

Interview nuance: if you claim "exactly-once delivery," expect a challenge. The correct framing: we get at-least-once delivery from the broker plus idempotent consumers (dedupe on a message id or use an idempotency key) so that reprocessing a duplicate has no effect. Kafka's "exactly-once" is at-least-once delivery combined with idempotent producers (a producer id plus sequence number so the broker drops duplicate appends) and transactional writes that tie the consume-process-produce cycle to an atomic offset commit.

Consumer scaling

Consumer groups: each partition is assigned to exactly one consumer in a group, so parallelism is capped at the partition count. Consumers track their position with committed offsets. When a consumer joins or dies, the group rebalances partition assignments. Two subtleties: commit the offset after processing (at-least-once) not before (which would be at-most-once and lose messages on crash), and backpressure is natural because a slow consumer just lags (its offset falls behind) rather than dropping data. A poison message that keeps failing goes to a dead-letter topic after N retries so it does not block the partition. Producers batch messages to trade latency for throughput.

producer --partition by key--> topic P0 [m0 m1 m2 ...]  (leader + ISR followers)
                               topic P1 [n0 n1 n2 ...]
consumer group G: P0 -> C1, P1 -> C2   (one partition per consumer)
   process msg -> commit offset  (at-least-once) ; dedupe by id -> exactly-once processing

Recap: model it as a partitioned append-only log with per-partition ordering, get durability from ISR replication and acks=all, offer at-least-once delivery plus idempotent consumers for exactly-once processing (never claim exactly-once delivery), and scale reads with consumer groups where parallelism equals partition count.