Module 9.1: Kafka Fundamentals
The log as a data structure: topics, partitions, offsets, consumer groups, lag, what at-most-once and at-least-once actually promise, and the same ideas at AWS scale with graded Kinesis shard math, all read with SQL over simulated broker and stream metadata.
The Log: Topics, Partitions, and Offsets
The append-only log as a data structure: topics split into partitions, immutable offsets inside a partition, retention that eats the head, and the key choice that decides whether one partition carries the whole load.
Consumer Groups and Measuring Lag
How a consumer group divides a topic's partitions, what the committed offset really points at, and the one subtraction that turns broker metadata into the lag number an on-call engineer is paged about.
Delivery Semantics: What Each Guarantee Costs
At-most-once and at-least-once are not settings, they are the order in which you commit and process. Derive both from crash-time metadata and name exactly which messages get lost and which get redelivered.
Kinesis: Shard Math, Firehose Buffers, and the Chooser
Size a stream with the arithmetic an interviewer actually asks for: two hard per-shard limits, the larger of the two, rounded up. Then why Firehose is near-real-time by construction, and how to pick between Streams, Firehose, and MSK.
Module 9.2: Change Data Capture
Debezium-shaped change events as a changelog table, and the discipline of applying one to a warehouse: log-order apply with upserts and tombstones, lsn-bounded checkpoints, and choosing where dedup lives, graded by running the script twice.
Anatomy of a Change Event
Read a Debezium-shaped changelog, tell log-based capture from query-based polling, and rebuild current state from the events alone.
Ordering and Checkpoints: Out-of-Order Arrival and the Bounded Apply
State what Kafka does and does not promise about order, catch inverted arrivals with LAG, and write the lsn-ordered, checkpoint-bounded apply that survives shuffled input and a rerun.
Where Dedup Lives: Write-Side versus Read-Side
Pick between absorbing redelivery at the write and absorbing it at the read, price what absorbing it nowhere costs, and probe for redeliveries that disagree.
Module 9.3: Stream Processing in SQL
The windowing half of streaming, written by hand so nothing stays hidden inside an operator: tumbling windows from integer division and hopping windows from a range join, then the split between event time and processing time with a watermark deciding what counts as late and what a straggler does to a window that already published a number, and finally the session-window drill at interview speed.
Tumbling and Hopping Windows
Bucket an event stream into tumbling windows with epoch arithmetic, then into overlapping hopping windows with a join against a window table.
Event Time, Watermarks, and Late Events
Separate event time from processing time, apply a watermark to decide what counts as late, and measure what stragglers do to an already-emitted window.
Session Windows: The Interview-Speed Sessionization Re-Drill
Rebuild the sessionization query unscaffolded against a 15-minute clock, and map it onto the stream engine's session-window operator.
Module 9.4: Choosing Batch vs Streaming
The decision framework interviewers actually reward, then freshness measured as an SLA over run metadata, closing the level with an unscaffolded SLA report.
Batch, Micro-Batch, or Streaming: The Decision
Pick the pipeline shape from the latency the consuming decision requires, then find the over-engineered products in a portfolio and price the mistake.
Freshness SLAs over Run Metadata
Measure the age of the data, not the health of the job, against a per-pipeline freshness contract, and catch the green run that wrote nothing.