Log Compaction, Retention & Tiered Storage

Retention decides what a topic is

The same append-only log behaves as a replayable event stream or as a queryable table depending entirely on how you retain it, and getting this wrong is how teams either blow up storage cost or lose the ability to rebuild state.

Two fundamentally different retention policies

Delete retention (time or size): keep records for a window, then delete whole old segments. retention.ms=604800000 keeps 7 days; retention.bytes caps total size. This makes a topic a stream: an immutable, time-bounded history you can replay within the window. Audit logs, clickstream, and event-sourcing event stores use this. The replay window is the retention window.

Log compaction (cleanup.policy=compact): instead of deleting by age, Kafka guarantees it retains at least the latest value for every key, garbage-collecting superseded older values in the background. This makes a topic a table/changelog: the log is the full edit history, but its compacted tail is the current state of every key. A "current user profile" topic keyed by user_id is the canonical case. A brand-new consumer reads the compacted topic from offset 0 and materializes the entire current state without a database: how Kafka Streams rebuilds a KTable and how CDC pipelines bootstrap read models.

Compacted topic keyed by user_id, before compaction:
  (u1,"A") (u2,"X") (u1,"B") (u3,"Q") (u1,"C") (u2,"Y")
After compaction keeps latest per key:
  (u3,"Q") (u1,"C") (u2,"Y")   <- current state of every user

Deletes in a compacted topic use a tombstone: a record with the key and a null value. Compaction keeps the tombstone long enough for all consumers to observe the deletion, then removes both the tombstone and all prior values for that key.

Interview nuance: GDPR/right-to-erasure collides with long retention. An immutable 7-year audit stream cannot literally delete one user's records without breaking immutability, so the standard pattern is crypto-shredding: encrypt per-subject data with a per-user key and delete the key to render the data unrecoverable, rather than mutating the log. On a compacted topic, a tombstone plus compaction does the erasure directly.

Tiered storage (KIP-405, GA) decouples retention cost from broker disk: hot recent segments stay on local broker SSD; cold older segments offload to object storage (S3, GCS) transparently, and consumers reading old offsets fetch from object storage automatically. This makes cheap long or effectively infinite retention viable: 7 years of audit data at S3 prices instead of years of broker SSD, and brokers rebalance faster.

The subtle correctness trap: your dedup/idempotency window must be at least as long as the replay/retention window. If you keep 7 days of events but your consumer only remembers processed ids for 24 hours, replaying day-6 events sails past the dedup memory and double-applies them. Retention and dedup must be sized together.

Recap: delete-retention makes a topic a replayable stream bounded by its window; log compaction keeps the latest value per key and makes a topic a rebuildable table/changelog (with tombstones for deletes); tiered storage puts cold segments in object storage for cheap long retention; GDPR erasure on immutable logs uses crypto-shredding or tombstones; and the dedup window must be at least the replay window or replays double-apply.