Kinesis: Shard Math, Firehose Buffers, and the Chooser
Size a stream with the arithmetic an interviewer actually asks for: two hard per-shard limits, the larger of the two, rounded up. Then why Firehose is near-real-time by construction, and how to pick between Streams, Firehose, and MSK.
A shard is a partition with a price list
Amazon Kinesis Data Streams is the managed log AWS offers, and it appears in roughly two thirds of AWS-flavored data engineering loops. Its shard is exactly the partition you met in the first lesson: an ordered, independently addressed slice of the stream. The difference is that a shard comes with published hard limits, so capacity stops being a hand wave and becomes arithmetic.
Per shard:
- Write: 1 MB per second, OR 1,000 records per second. Whichever you hit first is the ceiling.
- Read: 2 MB per second, shared by all standard consumers on that shard. Enhanced fan-out gives each registered consumer its own 2 MB per second instead of splitting one.
Both write limits are real and independent, which is the entire point of the exercise. A clickstream of 2 KB records at 5,000 per second is 10 MB per second, so bytes bind and you need 10 shards. A telemetry firehose of 250-byte records at 12,000 per second is only 3 MB per second, but it is 12,000 records per second, so record count binds and you need 12. Size for the larger of the two requirements, then round up, because you cannot rent a third of a shard.
Stage 1 of 4: The workload is two numbers, not one: 3 MB/s AND 12,000 records/s. Sizing on either alone is the classic miss.
Rounding up without CEIL
SQLite has no CEIL, which turns out to be useful, because the workaround is an idiom worth owning. To divide x by n and round UP using integer division only:
(x + n - 1) / n
So shards for throughput, with the rate expressed in KB per second and 1 MB taken as 1,000 KB, is (kb_per_sec + 999) / 1000, and shards for record count is (records_per_sec + 999) / 1000. Payments is the row that proves you need it: 1,200 KB per second divided by 1,000 is 1 under plain integer division, and a single shard would throttle a payment stream from the first minute. The correct answer is 2.
One warning about the operands. Integer division only happens when both sides are integers. If your rate is a REAL, 1200.0 / 1000 is 1.2 and the idiom silently stops rounding, so cast the rate to an integer first.
Firehose is a hose, not a queue
Amazon Data Firehose is frequently confused with Kinesis Data Streams, and the confusion is worth clearing up in one sentence: you do not read from Firehose. It is a delivery pipe that takes records and lands them in S3, Redshift, OpenSearch, Snowflake, or an Iceberg table, with an optional Lambda transform and dynamic partitioning on the way. There is no consumer, no offset, no replay.
Firehose buffers, and it flushes on size OR interval, whichever fills first. A typical setting is 5 MB or 300 seconds. That is what "near real time" actually means here: at high throughput the size trigger fires in seconds and delivery feels instant, while at low throughput you wait out the interval every time. The latency floor of a trickle stream is the buffer interval, and it is a buffering fact rather than a marketing claim.
The chooser, as a precedence ladder
Interviewers ask this one as a scenario and expect an ordered rule, not a list of features. Read the requirements top to bottom and stop at the first match.
- Existing code speaks the Kafka protocol, or the team is invested in the Kafka ecosystem, so use Amazon MSK.
- Otherwise, the workload needs replay of old records, or more than one independent application reads the stream, so use Kinesis Data Streams.
- Otherwise it is plain delivery into a sink with light transformation, so use Firehose, which has no consumers to write and no shards to size.
Stateful processing on top of any of these runs on Amazon Managed Service for Apache Flink. Its vocabulary, checkpointing and backpressure, shows up in the streaming-patterns module of this level.
Common mistake: sizing only on megabytes per second. Small records are the trap: a stream of 250-byte events can be well under 1 MB per second and still need a dozen shards, because the 1,000 records per second limit binds long before the byte limit does. Interviewers seed exactly this case, and checking only one constraint is the fastest way to fail the question.
Interview nuance: when you are asked "how many shards do you need", say the number and then say what would change it. Peak rather than average is the input, because a stream provisioned for the mean throttles every evening. Adding a safety factor of 20 to 30 percent for bursts is normal. And on-demand mode exists precisely so that you do not have to guess, at a higher per-GB price, which is a reasonable answer for a workload whose shape you do not know yet.
On a real platform this differs. Here you compute shard counts with SQL over a sizing table. In practice you would write the same arithmetic into a capacity spreadsheet or straight into infrastructure code, and then watch
WriteProvisionedThroughputExceededin CloudWatch to find out whether you were right. Idle provisioned shards bill by the hour whether you use them or not, which makes over-provisioning a slow leak rather than a safe default. The concepts transfer directly: an Azure Event Hubs throughput unit and a Microsoft Fabric Eventstream partition are the same sizing conversation with different nouns.
CREATE TABLE stream_workloads (
workload TEXT,
records_per_sec INTEGER,
avg_record_kb REAL, -- average payload size; 1 MB = 1000 KB throughout this lesson
consumer_count INTEGER, -- how many independent applications read the stream
needs_replay INTEGER, -- 1 = the team must re-read old records
needs_kafka_api INTEGER, -- 1 = existing code speaks the Kafka protocol
sink TEXT
);
INSERT INTO stream_workloads (workload, records_per_sec, avg_record_kb, consumer_count, needs_replay, needs_kafka_api, sink) VALUES
('clickstream', 5000, 2.00, 3, 0, 0, 's3'),
('iot_telemetry', 12000, 0.25, 2, 0, 0, 'lambda'),
('payments', 800, 1.50, 1, 1, 0, 'ledger_service'),
('audit_log', 200, 4.00, 1, 1, 0, 's3'),
('order_events', 6000, 1.25, 4, 1, 1, 'flink_app'),
('cdn_logs', 2500, 1.00, 1, 0, 0, 's3'),
('app_metrics', 5500, 0.50, 1, 0, 0, 'opensearch'),
('partner_feed', 12, 0.50, 1, 0, 0, 's3');-- Both shard constraints, side by side, for three workloads with three different binding limits.
SELECT workload,
records_per_sec,
avg_record_kb,
CAST(records_per_sec * avg_record_kb AS INTEGER) AS kb_per_sec,
(CAST(records_per_sec * avg_record_kb AS INTEGER) + 999) / 1000 AS shards_for_throughput,
(records_per_sec + 999) / 1000 AS shards_for_record_rate
FROM stream_workloads
WHERE workload IN ('clickstream', 'iot_telemetry', 'payments')
ORDER BY workload;Apply
Your turn
The task this lesson builds to.
Write a query that returns each workload with the shards it needs, as (workload, kb_per_sec, records_per_sec, shards_needed), most shards first and then by workload name, over stream_workloads(workload, records_per_sec, avg_record_kb, consumer_count, needs_replay, needs_kafka_api, sink).
kb_per_sec is records_per_sec * avg_record_kb cast to an integer. shards_needed is the LARGER of the two shard requirements, each rounded up: one shard carries 1,000 KB per second (take 1 MB as 1,000 KB) and one shard carries 1,000 records per second. Alias the columns exactly kb_per_sec and shards_needed.
3 hints and 1 automated check are waiting in the workspace.
Practice
Make it stick
A second problem on the same idea, plus 3 bonus drills.
Write a query that returns each workload's recommended streaming service, as (workload, recommended_service), ordered by workload name, over the same stream_workloads table.
Apply this precedence and stop at the first match: needs_kafka_api = 1 means 'msk'; otherwise needs_replay = 1 or more than one consumer means 'kinesis_streams'; otherwise 'firehose'. The order of the rules is the graded part, because a workload can match more than one of them. Alias the computed column exactly recommended_service.
1 automated check is waiting in the workspace.