Streaming / Real-Time Analytics Pipelines

A fight against exact counting and late data

A real-time analytics pipeline turns an unbounded stream of events into aggregates you can query within seconds: per-minute counts, unique visitors, top-K trending items. At billions of events per day (a few million per second at peak) the entire design is a fight against two things: the cost of exact counting, and the fact that events arrive late and out of order.

The backbone

Producers write events to a partitioned, replayable log: Kafka or Kinesis. Partition by a key that both spreads load and preserves the ordering you need (for example, item_id so all events for one item land on one partition in order). The log gives you three things a queue does not: replay (reprocess from an offset after a bug), backpressure (consumers pull at their own rate), and durability (retain days of data). Size it with the right rate for each question: the 2M/sec peak times 200 bytes is 400 MB/sec, which is what partition count has to absorb; retention is sized off the daily volume instead, and 5B events/day at 200 bytes is roughly 1 TB/day before replication. Peak rate and daily total are different capacity decisions, so never multiply a peak by 86,400 to get a day (that would claim 35 TB/day here, 35x the real figure).

The processing engine

A stream processor (Flink, or Spark Structured Streaming) consumes partitions and maintains windowed state. Windows come in three shapes: tumbling (fixed, non-overlapping, for per-minute counts), sliding (overlapping, for a trailing 5-minute top-K refreshed every 30s), and session (gap-defined, for user activity). The hard part is time. Event time (when it happened) differs from processing time (when you saw it). A phone offline for 10 minutes floods you with old events. Windows are keyed on event time, and a watermark is the engine's assertion that no event older than time T will still arrive. When the watermark passes a window's end, the window closes and emits. Late events past the watermark go to a side output or a small allowed-lateness update, never silently dropped.

Delivery semantics. At-least-once is cheap but double-counts on retry. Exactly-once needs the processor to checkpoint state and offsets atomically (Flink's distributed checkpoints) and sinks to be idempotent or transactional. For counts, exactly-once matters; for a fuzzy trending list, at-least-once with idempotent upserts is often enough.

Approximate structures, the core insight

Exact distinct counts and exact top-K over a firehose need unbounded memory (a set of every id seen). You trade a bounded error for bounded memory:

HyperLogLog  -> unique counts (cardinality) in ~12 KB per key, ~2% error
Count-Min Sketch -> per-item frequency in fixed memory, over-counts only
Top-K (heavy hitters, on top of CMS) -> trending items without a full sort
t-digest / DDSketch -> p50/p95/p99 latency quantiles in a tiny footprint

HyperLogLog also merges: per-partition sketches union into a global unique count, which is why it scales horizontally.

Serving, and Lambda vs Kappa

Do not query Flink state directly. Land aggregates in a real-time OLAP store built for high-ingest, sub-second aggregation: Apache Druid, Pinot, or ClickHouse. They pre-aggregate on ingest and answer "counts per minute for the last hour" in tens of milliseconds under dashboard concurrency.

Lambda runs a batch layer (exact, slow) alongside the speed layer (approximate, fast) and merges them, at the cost of two codebases. Kappa runs one streaming pipeline and reprocesses from the log by replaying when you need a correction. Kappa is the modern default because replay makes the batch layer redundant.

Interview nuance: when asked for "exact" trending, name the cost explicitly. Exact top-K needs a global count per item, which is a shuffle-heavy full aggregation. State that approximate top-K is a deliberate accuracy-for-scale trade, not a shortcut you forgot to fix.

Recap: Kafka backbone, Flink windows keyed on event time with watermarks for late data, exactly-once via checkpointing where counts must be right, HyperLogLog and Count-Min Sketch for bounded-memory counting, and a Druid/Pinot/ClickHouse serving layer for sub-second queries.