IoT / Edge Ingestion Architecture

A write-fan-in problem where you trust nothing

An IoT platform is a write-fan-in problem: a huge fleet of small devices each dribbles telemetry toward the cloud, and the platform must never assume a device is online, well-behaved, or trustworthy. With 10M devices each emitting one reading every 10 seconds you are already at 1M messages/sec sustained, and fleets are bursty (whole regions reconnect at once after an outage), so the design must absorb spikes several times the average.

The edge-cloud split

Push work to the edge when it cuts bandwidth or when latency matters for control. A smart thermostat should not stream raw 50Hz sensor data to the cloud; a gateway (a Raspberry Pi class device, or an on-prem box like AWS Greengrass / Azure IoT Edge) filters, aggregates ("send the 1-minute average, plus any reading outside a band"), and runs local inference so a safety cutoff fires in milliseconds without a cloud round trip. The cloud gets a compressed, pre-filtered stream instead of the firehose.

Protocols and offline buffering

Devices talk over lightweight protocols, not HTTP-per-reading. MQTT (a pub/sub broker protocol over a persistent TCP connection) dominates: one long-lived connection, tiny headers, QoS levels (0 fire-and-forget, 1 at-least-once, 2 exactly-once), and a "last will" message the broker publishes when a device drops. CoAP (UDP, REST-like) is used on the most constrained/low-power links. Crucially, devices buffer offline: when connectivity drops, the edge does store-and-forward, persisting readings locally and replaying them on reconnect. That means the cloud must accept late and out-of-order data and dedupe on a device-supplied event id.

devices --MQTT/CoAP--> edge gateway (filter, aggregate, buffer, local inference)
                                     |
                          MQTT broker cluster (auth, backpressure)
                                     |
                             ingest gateway --> Kafka (durable buffer)
                                    /                        \
                        hot path: stream alerting        cold path: batch -> lake/TSDB

Ingestion, hot/cold split, and control

The ingestion gateway sits behind the broker and does device provisioning and auth (each device gets its own X.509 certificate, never a shared key, so one compromised device can be revoked without re-keying the fleet), applies backpressure (reject or shed low-QoS traffic before the pipeline melts), and writes into a durable buffer like Kafka so a slow downstream consumer never blocks ingestion. From Kafka the stream forks: a hot path (Flink / Kafka Streams) evaluates alerting and anomaly rules in seconds, and a cold path lands raw data in S3 / a lake and a time-series DB for batch analytics and ML training.

Control flows the other way via a device shadow / digital twin: a cloud-side JSON document of each device's desired and reported state. You write the desired state, and the device reconciles when it next connects, which is exactly how OTA firmware rollouts work: stage to 1% (canary), watch crash/health telemetry, then ramp, so a bad image cannot brick 10M devices at once.

Interview nuance: the classic failure is assuming devices are always online. Without offline buffering you silently lose data during every outage; without dedupe you double-count the replay. And a thundering herd of reconnects after a regional outage can DDoS your own broker, so devices need randomized exponential backoff with jitter on reconnect, and the broker needs connection-rate limiting.

Recap: filter and buffer at the edge, connect over MQTT with per-device certs, absorb bursts and reconnects with a Kafka buffer and backpressure, fork into a seconds-latency hot path and a durable cold path, and drive control and OTA through a device shadow with canary rollout.