Why not call the email provider directly from the request that creates the order?

A slow or failing provider would slow or fail the order, and a failure would lose the email. With a queue, the request saves the event and returns at once, and workers send in the background and retry. The trade-off is a small delay and one more system to run.

A worker sends an email but the provider's reply times out. It retries. How do you avoid a duplicate?

Give each notification a unique idempotency key and send it with the request, and record the key in your own table. A repeat with the same key is ignored by your system and, where supported, by the provider. The trade-off is storing keys for a while, and some providers only offer best effort. A crash after sending but before recording the result can still cause a rare duplicate, so aim for very rare duplicates rather than none.

How should the retry policy work?

Retry temporary failures with exponential back-off and jitter, up to a maximum number of attempts. Do not retry permanent failures such as an invalid address. After the last attempt, move the message to a dead letter queue and alert. The trade-off is that very late delivery may be useless for some messages, so give messages an expiry.

A marketing campaign of five million emails starts. How do you keep one-time passcodes fast?

Use separate queues and worker pools for urgent and bulk traffic, and rate-limit the bulk queue to a steady pace. Passcodes never wait behind the campaign. The trade-off is more queues to monitor and capacity reserved for urgent traffic.

Your SMS provider is down for an hour. What should your system do?

Keep accepting events on the queue. A circuit breaker stops workers calling the failing provider, traffic fails over to a second provider where one is configured, and the rest waits and retries. Expire messages that are no longer useful, such as old passcodes. The trade-off is cost and complexity of a second provider against delay and lost messages.

Roughly how much load does a notification system for 10 million users face?

State the assumption first. With 5 notifications per user per day, that is 50 million a day, or about 580 per second on average. Assuming peaks are 10 times the average, plan for about 5,800 per second. At roughly 500 bytes per delivery row, the log grows about 25 GB a day. The trade-off is that the average hides the real constraints, which are the peak, provider rate limits and the backlog after an outage.

The provider recovers after an hour and 2 million messages are waiting. What goes wrong and what do you do?

Sending everything at once hits provider rate limits and delays fresh urgent messages behind old ones. Drain at a rate limit per provider, keep urgent traffic on its own queue, drop expired messages such as old passcodes, and add workers if the spare capacity is too small. The trade-off is that older messages arrive later, and some are dropped on purpose.

Notification System | TheLearningLoop

Why you need a notification system, and when to use one

Apps send emails, SMS messages and mobile push notifications for receipts, alerts, reminders and marketing. Each goes through a third-party provider that can be slow, down or rate-limited. If the code that handles a user request calls the provider directly, a provider outage fails the request or silently loses the message.

A notification system accepts events, decides who should get what on which channel, and delivers reliably in the background. It gives you two things interviewers always ask about:

Reliability: every important notification is delivered, even if a provider is down, and duplicates are kept as rare as possible. Exactly once is not fully achievable, as the section on delivery guarantees explains.
Scalability: a burst (a flash sale, a breaking alert) is spread over time instead of overwhelming providers or users.

Use one when:

Several services send notifications and they should share channels, templates and rules.
Messages must survive provider outages and be retried.
Users have preferences, quiet hours and unsubscribe choices to respect.
You must limit how often someone is contacted.

Skip it, or use something else, when:

One service sends a few emails: call the provider through a small queue or a library.
The message must be seen instantly in the page: use the live connection or polling instead of a channel like email.

Try it

Six emails must be sent. Change the design, then break the email provider.

Notification System

Step 1 of 11 · Press Space to play, arrow keys to step

DesignFailure

Initial state. The app must send 6 emails, one per tick. The provider is a third party that can fail.

tick:: 0
delivered:: 0
lost:: 0
duplicates:: 0
waiting in queue:: 0

1 tick = one step of the simulation.

Speed

Things to try:

Run Direct with Email provider is down. The emails sent during the outage are lost.
Run Queue + workers with the same failure. Retries with back-off deliver everything after the provider recovers.
Try the reply is lost, so we retry with each design. Direct sends duplicates, but the queue with an idempotency key does not.

What to watch (the provider is down for ticks 2 to 5, which is 4 ticks):

Direct, provider down: emails 2, 3, 4 and 5 are lost. Only emails 1 and 6 arrive, so the end counters read 2 delivered and 4 lost.
Queue, provider down: the waits before each retry grow (1, 2, then 4 ticks, and the simulation stops growing them at 4). Emails 3, 5 and 6 arrive at tick 6, email 4 at tick 7 and email 2 at tick 9. The end counters read 6 delivered, 0 lost and 0 waiting.
Direct, reply lost: email 2 arrives twice, so the counters show 1 duplicate.
Queue, reply lost: the retry of email 2 at tick 3 is ignored because it has the same key, so duplicates stay at 0.
Assumption to remember: the simulation lets the provider ignore a repeated key. Real providers do not all do this, as the next sections explain.

Requirements and rough numbers

Before choosing a design, state what it must do and how big it is.

Functional requirements: accept events from many services, choose the channel (email, SMS, push), fill in a template, respect each user's preferences and quiet hours, deliver, retry, and keep a delivery record.

Non-functional requirements: do not lose important messages, keep duplicates rare, send urgent messages such as one-time codes within seconds, absorb bursts without hurting providers, and stay up when a provider is down.

Illustrative sizing (assumed numbers, not measurements): 10 million users who each receive 5 notifications a day.

Quantity	Working	Result
Notifications per day	10,000,000 x 5	50 million
Average rate	50,000,000 / 86,400 seconds	about 580 per second
Peak rate if bursts are 10 times the average	580 x 10	about 5,800 per second
Delivery log per day if each row is about 500 bytes	50,000,000 x 500 bytes	about 25 GB
Backlog after a 1 hour outage at the average rate	580 x 3,600 seconds	about 2.1 million messages

The average is small. The peak, the provider limits and the backlog after an outage are what shape the design, which is why the queue exists.

Notification delivery strategies

Strategy	How it works	Good when	Weakness
Direct call	The request calls the provider and waits	Prototypes and rare messages	Slow requests, lost messages on failure
Queue and workers	Events go on a queue. Workers send in the background	Most production systems	Needs a queue and retry rules
Priority queues	Separate queues for urgent and bulk messages	Alerts and one-time codes mixed with marketing	More queues to run and monitor
Batching and digests	Combine many events into one message	Reduce noise, cut cost	Delay
Outbox plus queue	Write the event with the business data, publish it afterwards	Notifications must match committed data	A relay to run

At least once vs exactly once

Queues deliver at least once, and a timeout does not tell you if the provider sent the message. So a retry can send twice. Providers cannot do true exactly-once, so you use an idempotency key (such as the notification ID). Your own system checks it so a repeat is ignored, and the provider checks it too where it offers such a feature. The trade-off is storing keys for a while.

Be honest about the limits. Not every provider supports idempotency keys, so check the documentation of the one you use. Even your own check has a gap: if a worker sends a message and crashes before it records that it sent it, the retry sends it again. The goal is very rare duplicates, not zero. The simulation above assumes the provider ignores a repeated key, which is the best case.

How retries and back-off work

A worker takes a message and calls the provider.
On a temporary failure (a timeout, a server error, a rate limit) it puts the message back with a delay that grows each time (exponential back-off), often with some randomness (jitter).
After a maximum number of attempts, it moves the message to a dead letter queue for people or a separate process to look at.
On a permanent failure (an invalid address, an unsubscribed user) it stops at once and records why.
It records every outcome, so support can answer "did my email go out?"

Three details matter in practice. Cap the wait (for example at a few minutes) so it does not grow without limit. If a provider says how long to wait, such as a rate limit reply with a retry time, use that instead of your own guess. Give messages an expiry, because a one-time code that arrives an hour late is useless and should be dropped, not sent. Note that with a limit of five attempts there are only four waits between them, as the example below shows.

Notification system vs the alternatives

Concern	Queue-based service	Direct provider calls	Database polling job
Survives a provider outage	Yes	No	Yes
Keeps requests fast	Yes	No	Yes
Handles bursts	Yes	No	Slowly
Respects preferences in one place	Yes	Each service re-implements	Possible
Latency to send	Low when healthy	Lowest	Higher
Extra moving parts	Queue, workers	None	A job and a table
Best for	Production systems	Prototypes	Small systems

Use the queue-based service when notifications matter. Direct calls are fine for a prototype.

Avoiding a new single point of failure

Queue: use a durable, replicated queue so accepted events survive a crash.
Workers: run several. If one dies, its unacknowledged messages are redelivered once a timeout (often called a visibility timeout or lease) expires. Some queues do this for you. With Redis Streams another worker must claim the pending messages, and a database queue needs a lease column.
Providers: configure a second provider per channel and fail over when the first is unhealthy.
Preference and template services: cache them, and fall back to safe defaults.

Three ideas from the strategies table and the interview deserve a line each:

Outbox: if a service saves an order and then publishes an event, a crash between the two loses the event. The outbox pattern saves the event in the same database transaction as the order, and a relay publishes it afterwards. The relay may publish twice, so the idempotency key is still needed.
Circuit breaker: when a provider keeps failing, workers stop calling it for a short time instead of hammering it, then try one request to see if it has recovered.
Failover has a cost: if the first provider timed out, you cannot know whether it sent the message. Sending through the second provider may deliver twice, and an idempotency key at the first provider does not help. For one-time codes this is acceptable. For a receipt you may prefer to wait.

You accept a delay of seconds during failures, and the complexity of retries and duplicate handling.

Bottlenecks and scaling

Draining a backlog: after an outage, the queue holds everything that arrived meanwhile. If the workers can send 3,000 messages per second in total (an assumed figure) and live traffic is 580 per second, only 2,420 per second is spare. The 2.1 million message backlog from the sizing table then needs about 2,100,000 / 2,420, roughly 870 seconds or 14 minutes. During that time, fresh messages wait behind old ones, so urgent traffic needs its own queue.
Provider limits: providers limit how fast you may send. Workers need a rate limiter per provider, otherwise a backlog turns into a wave of rejected requests. For example, a bulk campaign of 5 million emails at 1,000 per second takes 5,000 seconds, about 83 minutes.
Scaling the queue: split queues by channel and priority, and split a busy queue into partitions so many workers read in parallel. Add workers when the queue depth keeps growing.
Per-user limits: a service may emit many events for one user. Apply a per-user rate limit or a digest before sending, not after.
Push notifications: a push needs a device token. Tokens expire or are revoked, so when the push service reports an invalid token, delete it. That is a permanent failure and should not be retried.
The delivery log: it grows with every message. Keep recent rows hot and archive or drop old ones.

Common mistakes

Sending inside the user's request. A slow provider slows the user.
Retrying without an idempotency key. Users get duplicates.
Infinite retries. A bad message loops for ever. Cap attempts and use a dead letter queue.
One queue for everything. A marketing burst delays a one-time password. Separate by priority.
Ignoring preferences and quiet hours. It annoys users and can break the anti-spam and data protection rules of the regions you send to. Check what applies to you.
No delivery record. Nobody can tell what was sent.

How to talk about a notification system in an interview

Ask which channels (push, SMS, email), what volume, and which messages are critical.
Draw services, a queue, per-channel workers, providers, a preferences store and a delivery log.
Cover retries with back-off, a dead letter queue, idempotency, priority and rate limits per user.
Name the trade-off: a queue gives reliability and smooth load at the cost of a small delay and more parts.
Start with requirements and one back-of-the-envelope estimate (events per day, average and peak per second) and say what the peak and the outage backlog mean for the number of workers.
Say where duplicates are stopped (your own store, the provider where it supports keys) and admit that a rare duplicate can still occur.

Where you meet notification systems in real systems

Every order confirmation, one-time code and "someone commented" alert comes from one. Here are three drawn out, the first with configuration you can run, then more examples.

Example 1: A durable notification queue with a retry-safe worker

Notifications are rows with a unique key, so a repeat is rejected. A worker claims due rows, sends them, and marks them sent or schedules a retry. A stream with a consumer group is the queue version. Open the example below to see the picture and the commands.

Config: notifications table and a stream queueSetup 1 table and 5 commandsCode no code changes

The event is saved with an idempotency key and queued. A worker sends it. If the provider fails, it is retried later with a longer wait, and after too many attempts it goes to a dead letter queue.

-- One row per notification. The unique key makes saving the same event twice a no-op.
CREATE TABLE notifications (
  id              bigserial PRIMARY KEY,
  idempotency_key text NOT NULL UNIQUE,
  channel         text NOT NULL,                 -- email, sms or push
  recipient       text NOT NULL,
  body            text NOT NULL,
  status          text NOT NULL DEFAULT 'pending', -- pending, sent or dead
  attempts        int  NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now()
);

-- Saving the same event again does nothing instead of creating a duplicate.
INSERT INTO notifications (idempotency_key, channel, recipient, body)
VALUES ('order-1001-receipt', 'email', 'user@example.com', 'Thanks for your order')
ON CONFLICT (idempotency_key) DO NOTHING;

-- A worker claims due notifications. SKIP LOCKED lets many workers run without clashing.
BEGIN;
SELECT id, channel, recipient, body FROM notifications
WHERE status = 'pending' AND next_attempt_at <= now()
ORDER BY id LIMIT 10
FOR UPDATE SKIP LOCKED;

-- After a failed send: count the attempt and wait longer each time (back-off, in seconds).
UPDATE notifications
SET attempts = attempts + 1,
    next_attempt_at = now() + (interval '1 second' * power(2, attempts + 1)),
    status = CASE WHEN attempts + 1 >= 5 THEN 'dead' ELSE 'pending' END
WHERE id = 1;
COMMIT;

The SQL was run against a real PostgreSQL 16 server: creating the table, a repeated insert doing nothing, claiming rows with SKIP LOCKED and a back-off update. The Redis commands were run against a real Redis 7 server: creating the group, adding, reading as a consumer, acknowledging and adding to the dead letter stream. Names and limits are examples.

Two things to know about this example. First, in the SQL the claim, the send and the update sit in one transaction for clarity. That keeps row locks open while the provider call is in flight. Many production systems instead mark the row as claimed with a lease time, commit, send, and then update the result. Second, if a worker crashes after sending but before it commits, the transaction rolls back and the message is sent again, which is why the idempotency key and a provider key where available both matter. In Redis Streams there is no built-in dead letter queue: the last command above is one way to build it by hand, and pending messages from a dead worker are recovered by another worker claiming them.

Example 2: Channels and priorities

Urgent messages such as a one-time code travel on a fast queue with their own workers. Bulk messages such as marketing use a slow queue so they never delay the urgent ones.

The notification service routes each message to a priority queue. Urgent workers send immediately. Bulk workers send at a gentle rate.

Example 3: Retry back-off

Waits grow so a recovering provider is not hit by a wall of retries.

The wait doubles after each failed attempt. After the last attempt the message moves to the dead letter queue.

With a limit of five attempts, the SQL above uses four waits: 2, 4, 8 and 16 seconds. The fifth failure sets the status to dead, so the 32 second wait in the table above would only apply if you allowed a sixth attempt. With jitter, each wait is randomised a little, so many failed messages do not all retry at the same instant. The simulation uses shorter waits (1, 2, 4 and 4 ticks) so that the whole run fits on screen.

More real-world scenarios

The scenarios below are illustrative patterns, not descriptions of any named company's system.

4. Order confirmations

An order event triggers an email and a push. The order ID plus the message type (for example order-1001-receipt) is the idempotency key, so a retry does not send a second receipt, while the shipping email for the same order still goes out. A rare duplicate is still possible if a worker crashes between sending and recording.

5. One-time passcodes

Codes use the urgent queue, expire in minutes, and fail over to a second SMS provider if the first is slow.

6. Social alerts

"Someone liked your post" events are batched into a digest when there are many, to avoid flooding a user's phone.

7. Marketing campaigns

A million emails are fed at a steady rate through the bulk queue so the provider's limits and the sender's reputation are protected.

8. Incident alerts

An on-call system pages people through push, SMS and phone calls in order, and escalates if nobody acknowledges.

9. Reminders and scheduled messages

Messages store a send time. A scheduler selects those that are due and puts them on the queue.

10. Preferences and unsubscribes

Before sending, the service checks the user's channel preferences, quiet hours and unsubscribe list in one place.

Tools and managed services

Kind	Name	Notes
Queue	Apache Kafka, RabbitMQ, Redis Streams	Hold notifications until workers send them
Managed queue	Amazon SQS, Google Cloud Pub/Sub, Azure Service Bus	Hosted queues with retry and dead letter support
Email and SMS	Amazon SES, Amazon SNS, Twilio, SendGrid	Third-party providers behind a common interface
Mobile push	Apple Push Notification service, Firebase Cloud Messaging	Deliver to iOS and Android devices
Database	PostgreSQL	Delivery log and `SKIP LOCKED` job tables

The takeaway

Whenever a design sends messages through a third party, put a queue between the event and the provider, retry with back-off, and give every message an idempotency key.

Notification System

Key takeaways

Why you need a notification system, and when to use one

Try it

Notification System

Requirements and rough numbers

Notification delivery strategies

At least once vs exactly once

How retries and back-off work

Notification system vs the alternatives

Avoiding a new single point of failure

Bottlenecks and scaling

Common mistakes

How to talk about a notification system in an interview

Where you meet notification systems in real systems

Example 1: A durable notification queue with a retry-safe worker

Example 2: Channels and priorities

Example 3: Retry back-off

More real-world scenarios

4. Order confirmations

5. One-time passcodes

6. Social alerts

7. Marketing campaigns

8. Incident alerts

9. Reminders and scheduled messages

10. Preferences and unsubscribes

Tools and managed services

The takeaway

Interview questions