Batch, Micro-Batch, or Streaming: The Decision
Pick the pipeline shape from the latency the consuming decision requires, then find the over-engineered products in a portfolio and price the mistake.
One question decides it
Every good batch-versus-streaming answer starts from the same question, and it is not a question about technology:
What decision consumes this data, and what latency does that decision require?
The consuming decision sets the clock. The clock picks the pipeline. Run the question on three products and it answers itself:
- A fraud model that has to block a card transaction before it settles has seconds. That is streaming, and the cost is earned.
- A storefront that has to stop showing an item that just sold out has minutes. Minutes is not streaming. Minutes is a job that runs every few minutes.
- A revenue review that happens Monday morning has a day. That is one scheduled run, and anything fancier is money spent on nothing.
Notice what the question is not. It is not "is streaming better", it is not "what does the team already run", and it is not "what sounds modern". Interviewers ask this because production teams ask it, and because the wrong answer is expensive in both dollars and pager volume.
The three shapes, and what each one costs you
Batch runs on a schedule, reads a bounded window of data, writes its output, and exits. One process, one input, one output, a clean success or failure, and a rerun you can reason about. Backfilling is just running it again with a different date parameter. It is the cheapest thing to build, the cheapest thing to run, and the easiest thing to debug at 3am.
Micro-batch is batch with a short period. The same bounded job runs every few minutes over whatever arrived since the last run. This is the entry juniors skip, and it is the most useful one on the list: it collects nearly all of streaming's perceived benefit at nearly none of streaming's cost. Most systems described as "real-time" in a product meeting are micro-batch underneath.
Streaming is a process that never exits. It reads an unbounded log, keeps state, emits results continuously, and has to answer questions the other two never face. When is a window complete. What happens to an event that arrives late. How does the job rebuild its state after a crash. Who gets paged when consumer lag climbs. Streaming is not a faster batch job, it is a different operational contract, and you sign that contract for the whole life of the pipeline.
Two clocks, and the ladder only reads one
Two numbers get mixed up constantly in this conversation, and separating them is most of what makes an answer sound senior.
Required freshness is the consumer's number: how old the data is allowed to be at the moment the decision is made. Run period is the pipeline's number: how often the job wakes up. The ladder below sets its boundaries on required freshness, because that is the number the consuming decision hands you, and it is the only one you cannot negotiate.
Run period is then a design choice underneath that boundary, and it has one rule: a job's period has to sit comfortably under the freshness it serves, because a job that runs every ten minutes can hand you data ten minutes stale on a bad day and cannot promise better. A job on a five to ten minute schedule serves a thirty-minute requirement with room for a retry. The same job cannot serve a three-minute requirement, which is why the ladder pushes anything under five minutes into streaming even though micro-batch periods that short exist.
If an earlier level left you with "micro-batch runs tiny batches every few seconds", replace it with the sentence above. A period measured in seconds is streaming's operational contract wearing a batch costume: at that cadence you are already paying for always-on infrastructure, state, and someone on call. Micro-batch worth the name is a scheduled job on a period of a few minutes, and that is what the 5-to-60-minute band is built for.
Under 5 minutes: streaming: ~1 min. A log plus a stream processor, always on. Highest cost, highest complexity: state, watermarks, replay, and an on-call rotation. Justified by fraud, real-time personalization, and operational systems.
The ladder is a CASE expression
The nice thing about a decision framework with numeric boundaries is that it is a query. A portfolio of data products, each with the freshness its consumer needs, classifies itself:
CASE
WHEN required_freshness_minutes < 5 THEN 'streaming'
WHEN required_freshness_minutes <= 60 THEN 'micro-batch'
ELSE 'batch'
END
The order of the branches is the whole trick. A CASE ladder stops at the first branch that matches, so writing the loosest condition first collapses every product into one bucket. Write the boundaries strictest to loosest and each branch only has to state its own upper limit.
Once the recommendation is a column, the gap between what a product needs and what it runs is a comparison, and the cost of that gap is already sitting in the next column. That comparison is the exact argument a junior engineer makes in an architecture review, and it is the graded work in this lesson.
Common mistake: defaulting to streaming because it sounds impressive. Interviewers name this one directly, and it is the single most reliable junior tell in a design round. Streaming buys latency and charges for it in state management, replay complexity, late-data handling, and on-call load, so proposing it for a dashboard that a human reads on Monday morning tells the interviewer you have never carried the pager for one.
Interview nuance: when a design prompt says "build a real-time pipeline", the strong-candidate move is to ask what decision consumes the output and how fresh it has to be before proposing an architecture. That single question separates candidates who have shipped pipelines from candidates who have read about them, and it is a cheap signal to send: it costs one sentence.
Two words worth having ready for the architecture round. Lambda architecture runs a batch layer and a speed layer in parallel and merges them at read time, which means two implementations of the same business logic and two chances to disagree. Kappa architecture keeps a single streaming path and rebuilds history by replaying the log, which is simpler to reason about but pushes everything into the streaming operational contract. Naming both, and naming the code-duplication cost of Lambda, is enough at the junior bar.
On a real platform this differs. Here the portfolio is one small table. On a real platform the same table exists as an architecture-review spreadsheet, a service catalog, or a set of tags on the pipelines themselves, and the cost column is joined in from a cloud billing export. The reasoning is identical and so is the query: classify by required freshness, compare against what is running, and sort the disagreements by money.
CREATE TABLE data_products (
product TEXT,
consuming_decision TEXT, -- the decision this data exists to support
required_freshness_minutes INTEGER, -- how old the data may be when that decision is made
pipeline_type TEXT, -- batch | micro-batch | streaming (what runs today)
monthly_cost_usd INTEGER
);
INSERT INTO data_products (product, consuming_decision, required_freshness_minutes, pipeline_type, monthly_cost_usd) VALUES
('fraud_scoring', 'block a card transaction before it settles', 1, 'streaming', 4800),
('ops_alerting', 'page on-call when ingestion stalls', 2, 'streaming', 2100),
('inventory_sync', 'hide an item the storefront just sold out of', 10, 'micro-batch', 900),
('support_routing', 'send a new ticket to the right agent queue', 15, 'batch', 800),
('ml_features', 'refresh the churn model feature table', 60, 'streaming', 3400),
('marketing_attribution', 'reallocate channel spend for tomorrow', 720, 'micro-batch', 1500),
('exec_dashboard', 'the Monday morning revenue review', 1440, 'streaming', 6200),
('finance_close', 'the month-end ledger reconciliation', 2880, 'batch', 700),
('partner_catalog', 'the weekly partner product feed', 10080, 'batch', 260);-- The architecture-review portfolio, strictest requirement first. The urgency column is a
-- plain CASE over required_freshness_minutes, phrased the way a stakeholder would say it.
SELECT product,
consuming_decision,
required_freshness_minutes,
CASE
WHEN required_freshness_minutes <= 2 THEN 'within a couple of minutes'
WHEN required_freshness_minutes <= 60 THEN 'within the hour'
WHEN required_freshness_minutes <= 1440 THEN 'within the day'
ELSE 'a few days is fine'
END AS urgency,
pipeline_type,
monthly_cost_usd
FROM data_products
ORDER BY required_freshness_minutes;Apply
Your turn
The task this lesson builds to.
Write a query that returns each data product with its recommended pipeline type, as (product, required_freshness_minutes, recommended_type), strictest requirement first, over data_products(product, consuming_decision, required_freshness_minutes, pipeline_type, monthly_cost_usd).
Use the framework's boundaries: under 5 minutes is streaming, 5 to 60 minutes inclusive is micro-batch, and over 60 minutes is batch. Alias the classified column exactly recommended_type, ordered by required_freshness_minutes ascending. Ignore the pipeline_type that is running today, this is the recommendation from first principles.
3 hints and 1 automated check are waiting in the workspace.
Practice
Make it stick
A second problem on the same idea, plus 2 bonus drills.
Write a query that returns the products whose running pipeline is stricter than their requirement needs, with the monthly cost of that mismatch, as (product, pipeline_type, recommended_type, monthly_cost_usd), most expensive first, over the same data_products table.
Strictness runs streaming above micro-batch above batch. Keep a product only when its running pipeline_type sits higher on that scale than the recommended_type the freshness boundaries give it. Use the same boundaries as the Apply: under 5 minutes streaming, 5 to 60 micro-batch, over 60 batch.
3 hints and 1 automated check are waiting in the workspace.