Management wants a live count of orders per minute and an exact end-of-day report. How do you build both?

Write orders to a replayable log. A stream job keeps the per-minute count for the live dashboard, and a batch job recomputes exact totals from the same log each night for the report. The trade-off is running two paths, which a single stream job with replay (the Kappa approach) can reduce.

Events can arrive minutes late. How does a stream job decide a window is finished?

It uses event time and a watermark, an estimate that events older than a certain time have arrived. A window closes when the watermark passes its end plus an allowed lateness. Later events are dropped, written to a side output, or fixed by a batch rerun. The trade-off is waiting longer for completeness against a faster answer.

A stream worker crashes halfway through. How do you avoid losing or double-counting events?

The job checkpoints its state and its position in the log. After a crash it restarts from the last checkpoint and replays from there. With idempotent or transactional output, a replayed event does not change the result twice. The trade-off is checkpoint overhead and a short recovery pause.

What are the Lambda and Kappa architectures?

Lambda runs a batch layer for accurate results and a stream layer for fresh results, and merges them when reading. Kappa uses only a stream job and recomputes by replaying the log. Lambda needs two code paths that can drift apart, while Kappa needs a long-lived, replayable log and a capable streaming engine.

A team wants to stream everything. Why might batch be a better fit?

If results are needed hourly or daily, batch is simpler to build, test, rerun and operate, and results are exact. Streaming adds state, late data and failure handling. The trade-off is freshness, so use streaming only for the results that need it.

Batch and Stream Processing

Why you need batch and stream processing, and when to use each

Raw data piles up: clicks, payments, sensor readings, logs. Turning it into counts, reports and decisions is data processing, and it comes in two styles. Batch processing collects data and runs a job over a large chunk at once, such as every night. Stream processing handles each event as it arrives, so answers are always up to date.

It gives you two things interviewers always ask about:

Freshness (latency): how soon after something happens the result is available.
Correctness and throughput: exact answers over huge volumes, even with late data and machine failures.

A few words used below, in plain terms:

Word	Meaning
Event time	When the thing actually happened (the click).
Processing time	When the system got to see it. A delay makes the two differ.
Window	A slice of event time (for example one minute) that results are grouped into.
Watermark	The job's estimate that "events older than this time have all arrived".
Checkpoint	A saved copy of the job's state and its position in the log, used to restart.
Idempotent write	Writing the same result twice has the same effect as writing it once.

This lesson assumes you know what a message queue or event log is. If not, read the message queues lesson first.

Use batch when:

Results are needed hourly, daily or on demand, such as reports and billing.
You need to reprocess all history after fixing a bug.
Accuracy matters more than speed, and volumes are very large.

Use streaming when:

Decisions must be made in seconds: fraud checks, alerts, live dashboards.
Data is an endless flow with no natural end.
Work is spread evenly over time, avoiding a nightly peak.

Skip both, or use something simpler, when:

A database query over the current data is fast enough.
Volumes are small: a script with a schedule is plenty.

Try it

Count the clicks in a three-minute window. Switch between processing styles and add a late event or a crash.

Batch and Stream Processing

Step 1 of 7 · Press Space to play, arrow keys to step

ProcessingFailure

Initial state. Count the clicks that happened in minutes 1 to 3. Events can arrive out of order and a late one is on the way.

tick:: 0
minute:: 0
answer available at minute:: -
clicks counted:: 0
events missed:: 0
events reprocessed:: 0

1 tick = one step of the simulation.

Speed

Things to try:

Run Stream with no failure. The answer is available in the first minutes and updates as events arrive.
Run Batch. The exact answer appears only when the job runs.
Choose An event arrives late with each. The batch counts it, but the stream's live count leaves it out.
Choose The worker crashes mid-way with each. The final count is unchanged, because the stream resumes from a checkpoint and the batch is simply rerun.

What to watch: compare the counters answer available at minute and events missed between the two modes. That is the whole trade-off: the stream answers early but may be incomplete, the batch answers late but is exact. In this simulation the stream job closes the window one minute after it ends, which stands in for a watermark with a short allowed lateness. The watermark and checkpoint themselves are not drawn; they are explained in the sections below.

Processing styles

Style	How it works	Good when	Weakness
Batch	A job reads a large stored dataset and writes results	Reports, billing, model training, reprocessing	Results are as old as the last run
Stream	A job processes each event as it arrives, keeping state	Alerts, live counts, fraud detection	Harder: late events, state, exactly-once
Micro-batch	Tiny batches every second or so	Near real time with simpler tooling	Latency of one interval
Lambda architecture	A batch layer for accuracy and a stream layer for speed, merged at read time	Both exact and fresh results are needed	Two code paths to write and keep consistent
Kappa architecture	One stream job, with the log replayed for reprocessing	You want a single code path	Needs a long-lived, replayable log

How a batch job runs (MapReduce style)

A batch job spreads a large stored dataset over many workers, in three steps. Word count is the classic example.

Map: each worker reads its own slice of the input and emits key and value pairs, such as ("checkout", 1) for each event.
Shuffle: the pairs are grouped so that every pair with the same key goes to the same worker.
Reduce: each worker combines the values for its keys, such as adding the ones to get a count per page.

If a worker fails, only its task is run again on another machine, because the input is stored and does not change. That is why batch failure handling is simple. The costs are the disk and network traffic of the shuffle and the delay until the whole job ends. Engines such as Apache Spark keep more data in memory, but the idea of map, shuffle and reduce is the same.

Batch vs stream

Batch sees the complete data for a period, so results are exact and easy to reason about. Reruns are safe because input does not change. The price is delay.
Stream gives low delay, but it must decide when a time window is complete even though events can arrive out of order or late. It uses a watermark, an estimate that "events older than this have all arrived". Anything later is dropped, sent to a side output, or used to correct the result.

How stream processing stays correct

Events are written to a durable, replayable log (for example Kafka), each with an event time.
A stream job reads the log, groups events into windows by event time (tumbling, sliding or session windows) and keeps state such as running counts.
The job saves checkpoints of its state and its position in the log.
After a crash, it restarts from the last checkpoint and replays from there, so no event is lost and, with transactional output or idempotent writes, none is counted twice.
A watermark closes windows, with an allowed lateness for stragglers. Very late data can be fixed later by a batch rerun.

Window types. A tumbling window is a fixed slice that does not overlap (each minute). A sliding window overlaps (the last five minutes, updated every minute) and suits moving averages. A session window groups events separated by less than a gap of inactivity, and suits user visits.

Delivery guarantees. The log and the job usually give at-least-once delivery: after a crash some events are processed again. Combined with an idempotent or transactional write, the visible result is as if each event were processed exactly once. Exactly-once is therefore a property of the whole pipeline, not of one component.

Micro-batch engines cut the stream into small batches every second or so. They are simpler to run but cannot answer faster than one interval. A side output is a separate place where the job sends events it cannot place in a window, such as very late ones, so they can be inspected or corrected.

Scaling the pipeline

Partition the log by key. Events with the same key (for example a user id) go to the same partition and stay in order. Different partitions are processed in parallel by different workers.
Add workers up to the number of partitions. More workers than partitions sit idle, so choose the partition count with growth in mind.
Watch for hot keys. One very busy key overloads a single worker while the others are idle. Spread it with a salted key or a pre-aggregation step.
Backpressure. If the job is slower than the incoming rate, lag grows. Scale out, slow the producers, or drop low-value data on purpose.
State size. Windows and counts per key live in the job's memory or a local store, and every checkpoint copies them. Large state makes checkpoints and recovery slower.

For example (illustrative numbers, not a benchmark): a service producing 20,000 events per second with 2 KB per event writes about 40 MB per second to the log. A single worker that handles 5,000 events per second would need at least 4 partitions and 4 workers, plus headroom for peaks, so 8 would be a reasonable first design. A one-minute window with 10 million distinct keys holds 10 million counters in state, which is worth checking against memory before choosing the design.

Processing design vs the alternatives

Concern	Stream job	Nightly batch job	Query the database
Result delay	Seconds	Hours	Immediate
Exact with late data	Needs care	Yes	Yes
Handles huge history	Yes	Yes	Slow
Simplicity	Hard	Easy	Easiest
Rerun after a bug	Replay the log	Rerun the job	Not applicable
Best for	Alerts and live views	Reports and training	Small data

Start with a batch job, and add streaming only for the results that really need to be fresh.

Avoiding a new single point of failure

The log: replicate it and keep enough history to replay.
The processing cluster: run several workers, checkpoint state to durable storage, and restart failed tasks automatically.
Output: write results idempotently, so a replay does not double-count.
Backlog: monitor the lag between the newest event and the one being processed, and alert when it grows.
Bad events: a malformed event that crashes the job on every replay will block the pipeline. Send it to a dead-letter side output and carry on.

You accept the cost of running a cluster and the discipline of designing for replays.

Common mistakes

Using processing time instead of event time. A delay moves events into the wrong window.
Ignoring late data. Decide how late is allowed and what happens after.
No checkpoints. A crash loses state or reprocesses everything.
Non-idempotent output. A replay writes the same result twice.
Choosing streaming when daily is fine. It is more work to build and run.
Two copies of the logic in Lambda. The batch and stream code drift apart.
Believing "exactly-once" is free. It needs checkpoints and idempotent or transactional output, and it has a cost.
Too few partitions. The parallelism ceiling is the partition count, and it is hard to change later.

How to talk about batch and stream processing in an interview

Ask how fresh the result must be and how much history there is.
Offer batch for accuracy and reprocessing, streaming for low delay, and say when you need both.
Mention a replayable log, event time and watermarks, checkpoints, and idempotent output.
Give rough numbers: events per second, bytes per event, and how many partitions and workers that implies.
Name the trade-off: streaming trades the simplicity and exactness of batch for freshness.

Where you meet batch and stream processing in real systems

Every analytics dashboard, fraud system and recommendation pipeline uses one or both. Here are three drawn out, the first with SQL you can run, then more examples.

Example 1: The same count as a batch query and a running total

A batch job counts clicks by window with one query over stored events. A stream job updates the counts as each event arrives. Open the example below to see the picture and the SQL.

Config: windowed counts in SQLSetup 1 table and 2 queriesCode no code changes

The batch path reads stored events later and computes exact results. The stream path updates results as events arrive and can be corrected by a later batch run.

-- Events carry the time they HAPPENED (event_time) and the time they ARRIVED (received_at).
CREATE TABLE click_events (
  id          bigserial PRIMARY KEY,
  event_time  timestamptz NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  page        text NOT NULL
);

INSERT INTO click_events (event_time, received_at, page) VALUES
  ('2026-01-01 10:00:10+00', '2026-01-01 10:00:11+00', '/home'),
  ('2026-01-01 10:00:50+00', '2026-01-01 10:00:51+00', '/home'),
  ('2026-01-01 10:01:20+00', '2026-01-01 10:01:21+00', '/pricing'),
  ('2026-01-01 10:00:55+00', '2026-01-01 10:03:00+00', '/home');   -- a late event: happened at 10:00 but arrived at 10:03

-- BATCH: count by the minute the click happened, over everything stored. The late event is included.
SELECT date_trunc('minute', event_time) AS minute, count(*) AS clicks
FROM click_events GROUP BY 1 ORDER BY 1;

-- STREAM-like result: what a job that had already closed the 10:00 window at 10:02 would have shown.
-- It counted only events that had arrived by then, so the late event is missing.
SELECT date_trunc('minute', event_time) AS minute, count(*) AS clicks
FROM click_events WHERE received_at <= '2026-01-01 10:02:00+00' GROUP BY 1 ORDER BY 1;

Note on the Python version: it is deliberately simplified. It sets the watermark to the arrival time minus the allowed lateness, so the window closes when the arrival time reaches the window end plus the allowed lateness. Real engines derive the watermark from the event times they have seen (for example the highest event time minus a bound) and treat allowed lateness as a separate extra wait. The second SQL query is likewise an approximation: it uses received_at to stand in for what the stream had seen at 10:02.

The SQL was run against a real PostgreSQL 16 server: the batch query counted the late event in the 10:00 minute, and the stream-like query did not. The Python script was run with Python 3: it counted two events in the first window and dropped the late one.

Example 2: A pipeline with a replayable log

The log is the source of truth. The stream job and the batch job both read from it, and either can be replayed.

Applications write events to a replayable log. A stream job keeps live results, and a batch job recomputes exact results from the same log.

Example 3: Windows

Events are grouped into windows by the time they happened.

Each window collects the events that happened in its minute. A late event belongs in an earlier window, which is why the stream needs a watermark and a lateness rule.

Example 4: Lambda and Kappa

Lambda keeps a batch layer for exact results and a stream layer for fresh ones. A reader merges the two, which means two code paths to keep in agreement.

Kappa has a single stream job. To fix a bug or change the logic, start a new job version and replay the log from the beginning, then switch readers to its output.

Example 5: Window types

Events at seconds 10, 40, 70 and 200. Tumbling one-minute windows give three groups. A sliding two-minute window advancing every minute counts events in overlapping spans. A session window with a 60-second gap groups 10, 40 and 70 together, and 200 starts a new session.

More real-world scenarios

These are illustrative scenarios, not descriptions of any particular company's system.

4. Daily sales reports

A batch job totals the day's orders overnight, and the exact numbers go to finance.

5. Fraud detection

A stream job checks each payment against recent activity within milliseconds and blocks suspicious ones.

6. Live dashboards

Counts of visitors, errors or orders update every few seconds from a stream.

7. Recommendation models

A batch job retrains a model each night on the full history, while a stream updates a user's recent activity.

8. Log analysis and alerting

A stream job counts errors per service per minute and raises an alert when a threshold is passed.

9. Data warehouse loading

Batch jobs copy and transform data from operational databases into a warehouse on a schedule.

10. Billing and metering

Usage events are streamed in, windows are closed with allowed lateness, and a final batch run produces the invoice.

Tools and managed services

Kind	Name	Notes
Log	Apache Kafka	Durable, replayable event log
Stream processing	Apache Flink, Kafka Streams, Apache Spark Structured Streaming	Windows, state and checkpoints
Batch processing	Apache Spark, Hadoop MapReduce	Large scale jobs over stored data
Orchestration	Apache Airflow	Schedule and monitor batch jobs
Managed	Amazon Kinesis, Google Cloud Dataflow, Azure Stream Analytics	Hosted streaming

The takeaway

Whenever results come from a flow of events, keep the events in a replayable log, use batch for exact answers and reprocessing, and add a stream only where freshness matters.

Batch and Stream Processing

Key takeaways

Why you need batch and stream processing, and when to use each

Try it

Batch and Stream Processing

Processing styles

How a batch job runs (MapReduce style)

Batch vs stream

How stream processing stays correct

Scaling the pipeline

Processing design vs the alternatives

Avoiding a new single point of failure

Common mistakes

How to talk about batch and stream processing in an interview

Where you meet batch and stream processing in real systems

Example 1: The same count as a batch query and a running total

Example 2: A pipeline with a replayable log

Example 3: Windows

Example 4: Lambda and Kappa

Example 5: Window types

More real-world scenarios

4. Daily sales reports

5. Fraud detection

6. Live dashboards

7. Recommendation models

8. Log analysis and alerting

9. Data warehouse loading

10. Billing and metering

Tools and managed services

The takeaway

Interview questions