Design an Ad Click Aggregator / Real-Time Analytics

Fast and eventually exact

An ad click aggregator ingests a high-volume stream of click events and produces per-campaign counts that advertisers see in near real time and that also feed billing, so the numbers must be both fast and eventually exact. This is the canonical streaming-aggregation interview, and it lives or dies on two ideas: idempotent counting and reconciling real-time with batch truth.

Idempotent counting

The naive design fails immediately. If you just do counter++ per event on an at-least-once stream (Kafka redelivers on consumer restart), you double-count, and since clicks are money, that is fraud-by-bug. You need exactly-once or idempotent counting. Each click carries a unique id; dedup on it. At high volume you cannot keep every id forever, so use a bloom filter or a windowed dedup store (recent ids in Redis with TTL) to reject replays cheaply, accepting a tiny false-positive rate. Alternatively, lean on the stream processor's exactly-once semantics (Flink checkpointing, Kafka transactions) so an aggregate update and the source offset commit are atomic, meaning a replay after crash does not double-apply.

Interview nuance: state the delivery-semantics problem out loud: Kafka gives at-least-once by default, so naive increments double-count. Name your fix (Flink exactly-once via checkpointed state + transactional sink, or explicit dedup on click id), because "just increment a counter" is the failing answer.

Event time and watermarks

Clicks arrive late and out of order (a mobile device offline for an hour uploads its clicks later). You aggregate over windows (per-minute, per-hour tumbling windows per campaign), and you need watermarks to decide when a window is "done." A watermark is the stream's assertion that "no events older than T will still arrive," so the window can close and emit. You also configure allowed lateness: hold windows open a bit past the watermark to admit stragglers, and emit late updates for clicks arriving after close. Event time (when the click happened) not processing time (when you saw it) is what you window on, or your counts are wrong whenever ingestion lags.

Lambda / Kappa

Real-time systems are approximate and can have gaps, so the industry pattern is Lambda or Kappa. Lambda runs two paths: a fast streaming path (Flink) that gives immediate, slightly-approximate counts for the advertiser dashboard, and a slow batch path (Spark over the raw event log in S3, run hourly/daily) that recomputes the exact, deduplicated, fraud-filtered numbers that billing uses. The batch layer is the source of truth and corrects any streaming drift. Kappa simplifies to one streaming engine with replay: the same Flink job can reprocess from the Kafka/log retention to recompute, avoiding two codebases.

clicks -> Kafka (raw log, retained) --> Flink (windows + watermarks + dedup) --> sharded counters -> dashboard (fast, ~approx)
                          \--> S3 raw --> Spark batch (hourly, exact, fraud-filtered) --> billing (truth)

Hot campaigns create counter hotspots; a viral ad might take millions of increments/sec on one key. Shard the counter into N sub-counters updated independently and summed on read, and pre-aggregate within the stream processor before writing. Fraud/bot filtering (dedup, rate anomalies, click-farm patterns) runs in-stream for fast defense and again in batch for the authoritative purge.

Recap: dedup clicks idempotently (bloom/windowed store or Flink exactly-once) so at-least-once delivery does not double-count, window on event time with watermarks and allowed lateness for out-of-order clicks, use Lambda/Kappa so a fast approximate stream is reconciled by an exact batch (or replayable) source of truth, and shard hot-campaign counters.