Design a Metrics & Monitoring System (Prometheus/Datadog)

Write throughput and cardinality control

A metrics platform ingests a firehose of numbers over time (millions of data points per second from thousands of hosts), stores them cheaply, serves fast dashboard queries, and fires alerts. The interview is really about two things: write throughput into a time-series database, and controlling cardinality so cost does not explode.

The cardinality trap

A metric is a name plus a set of labels plus a timestamped value: http_requests_total{service="checkout", region="us-east", status="200"} = 4823 @ t. The unique combination of label values is a time series. Here is the trap that dominates this problem: cardinality is the product of all label value counts. Add a user_id label with 10M values and one metric becomes 10M time series, and your storage and query cost explode. Controlling cardinality (never put unbounded-cardinality fields like user id, request id, or email in labels) is the single most important design discipline.

Interview nuance: when asked "what breaks first," say high-cardinality labels. Interviewers want to hear that you would reject user_id/trace_id as labels, cap label sets, and detect cardinality spikes, because unbounded cardinality is what actually takes these systems down.

Ingestion and storage

Agents on each host batch and push samples (or the platform scrapes /metrics endpoints on an interval, the Prometheus pull model). A high-throughput front door (a stateless ingestion tier writing to Kafka) buffers the firehose and decouples spiky producers from storage. Batching and compression are essential: time-series data compresses beautifully because timestamps are regular and adjacent values are similar (delta-of-delta timestamp encoding plus XOR float compression, the Gorilla/Facebook technique, gets ~1.3 bytes per sample versus 16 raw).

Storage is a purpose-built TSDB (Prometheus TSDB, Cortex/Mimir, InfluxDB, TimescaleDB) organized for the dominant query pattern: "give me one series over a time range." Data is partitioned by time into blocks (recent blocks in memory/SSD for fast writes and hot reads, older blocks flushed to object storage) and indexed by label so a query can find matching series quickly.

Retention, rollups, alerting

You do not keep raw 1-second resolution for a year. Downsample: keep raw for a short window (e.g., 15 days), then pre-aggregate into 5-minute and 1-hour rollups (min/max/avg/count) for longer retention. A dashboard showing last quarter reads cheap hourly rollups, not billions of raw points. Retention tiers plus rollups are the cost-control lever alongside cardinality.

hosts -> agents (batch, compress) -> Kafka -> ingester (TSDB write)
                                                  |
   raw (hot, in-mem/SSD) --downsample--> 5m/1h rollups (cold, object store)
                                                  |
   Query engine (label index, range scan) -> dashboards
   Rule evaluator (every 15s) -> alerts -> dedup/group -> notify (PagerDuty/Slack)

Alerting is periodic rule evaluation. A rule engine runs queries on a schedule (e.g., every 15s), avg(rate(errors[5m])) > 0.05, and on a firing condition creates an alert. Crucially, an alert manager deduplicates and groups (one incident, not 500 pages from 500 hosts), applies silences/inhibitions, and routes to PagerDuty/Slack/email.

Recap: buffer the ingestion firehose through Kafka into a compressed TSDB partitioned by time, control cost with cardinality limits plus retention tiers and downsampled rollups, serve dashboards from a label-indexed query engine, and evaluate alert rules on a schedule with a dedup/group/route alert manager.