Batch vs Streaming: Lambda vs Kappa

One processing path or two?

The last piece is how data is processed over time, and the central interview question is whether you need two processing paths or one. Getting this right saves you from maintaining two codebases that slowly disagree.

Batch vs streaming

A throughput-versus-latency tradeoff. Batch processes a bounded chunk (yesterday's events) on a schedule: high throughput, simple correctness (you have all the data before you compute), high latency (results are hours old). Spark and classic MapReduce are batch engines. Streaming processes an unbounded flow event by event: low latency (seconds), continuous, but correctness is harder because data arrives late, out of order, and you must decide when a window is "done." Flink and Spark Structured Streaming are streaming engines, fed by a durable log (Kafka, Pulsar).

Lambda architecture

The first mainstream answer to "I need both fast and correct." It runs two parallel layers: a batch layer that reprocesses all history nightly to produce accurate, complete results, and a speed layer that processes the live stream for low-latency approximate results, with a serving layer merging the two so recent data comes from the speed layer and older data from the batch layer. It works and is self-correcting (the batch layer eventually overwrites any speed-layer approximation). The cost is brutal: you implement the same business logic twice, once in a batch engine and once in a streaming engine, in different code, and they drift. Every metric change is two implementations to keep in sync.

Kappa architecture

The reaction: delete the batch layer. There is one streaming path, and the durable log (Kafka) is the system of record with long retention. If you need to recompute history (bug fix, new metric), you replay the log from the beginning through the same streaming code. One codebase, one set of logic, no drift. Kappa is the default for new systems when the streaming engine can express your logic and the log retention is affordable.

  Lambda                          Kappa
  events -> batch layer  \        events -> Kafka (retained) -> stream job -> serving
         -> speed layer  -> serve                   ^                |
  (two codebases, merged)                            +-- replay to recompute

Event-time, watermarks, and delivery

Processing-time is when your job sees an event; event-time is when it actually happened. A phone offline in a tunnel sends events with an event-time from 10 minutes ago. If you window by processing-time you put those events in the wrong bucket and your per-minute counts are wrong. So you window by event-time, and a watermark is the engine's assertion "I believe I have now seen all events up to time T," which lets it close the window for T and emit results. Late events arriving after the watermark are handled by policy: drop them, or emit an updated result (allowed lateness). Watermarks are the explicit tradeoff between latency (advance aggressively, emit fast, risk dropping late data) and completeness (wait longer, more correct, higher latency).

Delivery semantics. At-least-once can double-count; exactly-once requires the engine to coordinate checkpoints with idempotent/transactional sinks. Flink provides exactly-once via distributed checkpointing (Chandy-Lamport) plus two-phase-commit sinks. For a fraud counter or a financial total this matters; for a rough traffic dashboard at-least-once is fine.

Streaming-into-lakehouse collapses the two paths

The modern move that makes Kappa practical for reporting too: Flink writes the stream directly into Iceberg tables (exactly-once). Now the live stream powers the real-time signal, and the same Iceberg tables it lands in are queried by batch SQL (Trino, Spark) for nightly reports. One pipeline feeds both the real-time consumer and the reporting consumer, so you no longer maintain a separate batch path at all.

Interview nuance: do not reflexively say "Lambda" because you need both real-time and batch outputs. State the condition: Lambda is justified only when the batch engine can express something the stream cannot, or when you need a periodic full-reprocessing guarantee the stream cannot give cheaply. Otherwise Kappa plus log replay plus streaming-into-lakehouse gives you both outputs from one codebase, and that is the stronger default answer.

Recap: batch is high-throughput/high-latency and simple, streaming is low-latency/continuous and correctness-hard; Lambda runs parallel batch and speed layers (accurate but two codebases that drift), Kappa runs one streaming path and replays the retained log to recompute; window by event-time with watermarks to handle late/out-of-order data trading latency for completeness; choose exactly-once where counts must be exact; and Flink-into-Iceberg collapses real-time and reporting into one pipeline.