How would you deliver messages in real time, and why not just poll?

Use WebSockets so the server can push a message as soon as it is saved. Polling adds delay up to the poll interval and sends many empty requests. The trade-off is that WebSockets keep millions of connections open, which needs connection-aware servers and careful limits.

Ana and Ben are connected to different chat servers. How does Ana's message reach Ben?

Keep a session store that maps each user to the server holding their connection, or publish on a channel that every server listens to. Ana's server saves the message, looks up Ben's server and forwards it there. The trade-off is an extra lookup per message, and the store itself must be replicated.

Ben is offline when Ana sends a message. What happens?

The message is already saved. The service sends a push notification through the phone's push provider. When Ben reconnects, his app asks for all messages after the last ID it has and receives them in order. The trade-off is that push is not guaranteed, so the stored copy is the real delivery.

How do you keep messages in a conversation in the right order?

Do not rely on timestamps from different machines. Route all messages of a conversation to one partition and give each a sequence number there, or use IDs that increase within the conversation. Clients display by sequence number. The trade-off is that one conversation is limited to one partition's write rate, which is fine for chat.

A chat server crashes with 100,000 connected users. What happens to them?

Their connections drop. Clients reconnect with back-off and jitter so they do not all hit the load balancer at once, land on other servers, and sync from their last message ID. No messages were lost because each was saved before delivery. The trade-off is a short delay and a reconnect storm that you must plan for. A message Ana sent while her server was down was never acknowledged, so her app keeps it and retries with the same client ID, and the de-duplication makes that safe.

How would you estimate the scale of a chat system?

State assumptions, then calculate. Illustrative: 100 million daily users sending 40 messages each is 4 billion a day, about 46,300 a second on average and about 139,000 at 3 times peak. At 200 bytes each that is about 800 GB a day. If 10 percent are connected at once and a server holds 100,000 connections, you need about 100 chat servers. Connections decide the server count, not message rate. The trade-off is that the assumptions are rough, so you confirm them with load tests.

Chat System | TheLearningLoop

Why a chat system is a classic design question, and what it teaches

A chat system lets people send messages and receive them within a moment, on any device, even if one end was offline when the message was sent. It combines real-time delivery, durable storage, ordering, presence and fan-out to many devices.

It gives you two things interviewers always ask about:

Low latency: a message should appear almost at once, so the server must push it rather than wait to be asked.
Durability and availability: no message is lost when a server crashes or a user is offline, and chat keeps working when one server fails.

Requirements to confirm first:

Functional: one-to-one and group chat, online and offline delivery, message history, delivery and read receipts, optionally media.
Non-functional: low delivery delay, no lost messages, messages in order within a conversation, millions of simultaneous connections.

Skip the design, or simplify it, when:

Messages can arrive minutes late: ordinary requests and a database are enough.
Only a few users are online: one server and one database are enough.

Rough capacity estimation

These numbers are illustrative assumptions, not measurements of any real service. Change them for your own case and keep the method.

Quantity	Assumption or calculation	Result
Daily active users	Assumed	100 million
Messages per user per day	Assumed	40
Messages per day	100 million x 40	4 billion
Average messages per second	4 billion / 86,400 seconds	about 46,300
Peak messages per second	Assumed 3 times the average	about 139,000
Storage per day	4 billion x 200 bytes (assumed size with metadata)	about 800 GB
Storage per year	800 GB x 365	about 292 TB, or about 876 TB with 3 copies
Users connected at once	Assumed 10 percent of 100 million	10 million
Chat servers	10 million / 100,000 connections per server (assumed)	about 100, plan for about 150 with spare capacity
Heartbeats	10 million connections, one ping every 30 seconds	about 333,000 per second

Two things the numbers teach. Connections, not message rate, decide how many chat servers you need. And the heartbeats that keep connections alive are about seven times the average message rate, so they must be cheap to answer.

Try it

Ana sends three messages to Ben. Change how Ben receives them, then break something.

Chat System

Step 1 of 14 · Press Space to play, arrow keys to step

DeliveryFailure

Initial state. Ana will send three messages to Ben. Every message is saved to the database before anything else.

tick:: 0
requests:: 0
delivered:: 0
avg delay (ticks):: -

1 tick = one step of the simulation.

Speed

Things to try:

Run WebSocket with no failure. Each message is pushed the moment it is saved, with no waiting for a poll and no empty requests.
Switch to Polling. Messages wait for the next poll, and some polls return nothing.
Choose Recipient is offline with each design. All three messages still arrive, because each one was saved before delivery was attempted.
Choose Chat server crashes with each design. Messages 1 and 2 were saved before the crash. Message 4 is sent while the server is down, so it is not saved yet: Ana's app keeps it and retries after the server returns. Nothing is lost because the app treats a message as sent only once the server acknowledges it.

What to watch (average delay in ticks, from the counters, for this three-message run):

Failure	WebSocket	Polling
None	0.0	1.7
Recipient is offline	1.3	6.7
Chat server crashes	0.7	3.7

The requests counter counts every push as well as every poll, so WebSocket shows 3 and polling shows 4 with no failure. The real saving is that polling's last two polls (ticks 9 and 12) return nothing, and every polled message waits for the next poll.

Ways to deliver messages

Approach	How it works	Good when	Weakness
Short polling	The client asks for new messages every few seconds	Simple, rare updates	Delay up to the interval, and many empty requests
Long polling	The server holds a request open until there is news	WebSockets are blocked	Every message costs a new request and a held connection
WebSocket	One two-way connection stays open and either side can send	Real-time chat	Millions of open connections to manage
Server-sent events	The server streams events one way	Notifications, live feeds	Cannot send from client over the same stream
Push notification	The phone operating system wakes the app	The user is offline or the app is closed	Not guaranteed or instant, and needs a provider

One chat server vs many

One server knows every connected user, so delivery is a memory lookup. It cannot hold millions of connections or survive its own failure.
Many servers each hold part of the connections. To send to Ben, the sender's server must find which server Ben is connected to, through a shared session store or a pub/sub channel, and forward the message there.

High-level design

Component	What it does	What it does not do
Client app	Opens the WebSocket, makes the message ID, retries until acknowledged, remembers the last message ID it has	Decide the order of messages
Load balancer	Spreads new connections over the chat servers, including reconnects	Route individual messages
Chat server	Holds open connections, saves each message, acknowledges it, pushes to its own connections and forwards to other servers	Keep the only copy of anything
Session store	Maps each user to the chat server holding their connection; can be rebuilt from reconnects	Hold messages
Pub/sub channel	Carries "deliver this to user X" between chat servers when you do not look up the server first	Remember a message nobody was listening for
Message store	Keeps every message in order per conversation and is the truth	Know who is online
Push service	Asks the phone's push provider to wake a closed app	Guarantee delivery
Presence service	Tracks heartbeats and last seen	Carry messages

The request path is: Ana, load balancer, her chat server, message store, session store lookup, Ben's chat server, Ben. If Ben is offline, the path ends at the push service instead.

How message delivery works

Ana's app sends the message over its WebSocket to her chat server, with a client-made ID.
The server saves it first (database or log) and acknowledges to Ana.
It looks up where Ben is connected. If he is online, it pushes the message to that server, which sends it to his device.
If Ben is offline, the message stays stored and a push notification is sent. When he reconnects, his app asks for everything after the last message ID it has.
Ben's app acknowledges receipt, which produces the "delivered" tick.

Because the stored copy is the truth and each message has an ID, a retry never creates a duplicate: the server ignores an ID it has seen. This holds when the check for a seen ID and the save happen as one atomic step, scoped to the sender and the client-made ID. Delivery on the wire is therefore at least once (a retry may arrive twice), and the ID check makes the visible result exactly once. That is what idempotent means: repeating the request has the same effect as doing it once.

Chat design vs the alternatives

Concern	WebSocket chat service	Polling an API	Email-style store and forward
Delivery delay	Near zero	Up to the poll interval	Minutes
Requests when idle	No message requests (only small heartbeats)	Constant	None
Works when the user is offline	Yes, with storage and push	Yes	Yes
Connections to manage	Many	Few	None
Complexity	High	Low	Low
Best for	Real-time conversations	Prototypes	Asynchronous messages

Build the WebSocket service when conversations are real time. Polling is enough for a prototype.

Avoiding a new single point of failure

Chat servers: keep them stateless apart from open connections, so a client that loses its server reconnects to another one through the load balancer and syncs from its last message ID.
Session store and pub/sub: replicate them, and rebuild the map from reconnects if it is lost.
Message store: replicate it and partition by conversation, so one conversation's messages live together and stay in order.

You accept a short reconnect delay when a server fails and some complexity around ordering.

Bottlenecks and how to relieve them

Open connections per server. Memory and file handles cap a server. The 100,000 per server in the estimate is an assumption to load test, not a limit. Use a least-connections rule on the load balancer, and when deploying, drain a server gradually so its clients reconnect in small groups.
Reconnect storm. When a server holding 100,000 connections dies, all its clients reconnect at once. Back-off (wait longer after each failed attempt) plus jitter (a random extra wait) spreads them out.
Hot partitions. Partitioning by conversation keeps order, but one very busy conversation lives on one partition. That is fine for ordinary chats. The real cost in a large group is fan-out, copying one message to many recipients, which is why very large groups pull for quiet members.
Presence heartbeats. In the estimate they are about seven times the average message rate. Answer them in the chat server's memory and update the shared presence store only when status changes.
Session store lookups. One lookup per message. Cache recent answers, replicate the store, and rebuild it from reconnects if it is lost.

The trade-off throughout: you pay in operational care (draining, back-off, load testing) to keep the low latency that open connections give.

Common mistakes

Sending before saving. A crash loses the message. Save, then deliver.
Ordering by arrival time. Different servers disagree. Use a per-conversation sequence number.
No client-generated ID. Retries create duplicates.
Treating connection count as the only limit. Fan-out in large groups needs its own plan.
Forgetting presence cost. Broadcasting every status change to every contact does not scale.
No sync on reconnect. The client must ask for messages after its last ID.

How to talk about a chat system in an interview

Confirm one-to-one vs group, history, receipts and the scale of simultaneous users.
Choose WebSockets for delivery and explain why polling does not fit.
Walk through save first, then deliver, how you find Ben's server, and what happens when he is offline.
Mention ordering per conversation, idempotent retries, push notifications, and the trade-off: holding many open connections buys low latency at the cost of operational complexity.

Where you meet chat systems in real systems

Messaging apps, customer-support widgets and in-game chat all follow this shape. Here are three drawn out, the first with configuration you can run, then more examples.

Example 1: Accepting WebSocket connections and storing messages

NGINX keeps long-lived WebSocket connections open to the chat servers. Redis Streams keep a durable, ordered log of each conversation and let a reconnecting client ask for everything after the last ID it saw. Open the example below to see the picture and the configuration.

Config: WebSocket proxy and a message logSetup 1 config and 4 commandsCode no code changes

The sender's server saves the message to the log, then pushes it to the recipient's server. A reconnecting client reads the log after its last known ID.

events {}
http {
  # Tell NGINX to upgrade the connection to a WebSocket when the client asks.
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }
  upstream chat {
    # Placeholder servers. Replace with your chat servers.
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
  }
  server {
    listen 80;
    location /ws {
      proxy_pass http://chat;
      proxy_http_version 1.1;                      # WebSockets need HTTP/1.1
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_read_timeout 3600s;                    # keep idle connections open; clients send pings
    }
  }
}

The NGINX file passed nginx -t on the official image. The Redis commands were run against a real Redis 7 server: adding two messages to a stream, reading them back in order, and publishing to a channel. Names and addresses are examples.

In the Redis block, XRANGE chat:room1 - + reads the whole log, as a client with no history would. A reconnecting client passes its last ID as an exclusive start instead, written XRANGE chat:room1 (last-id +, where the opening bracket means "after this ID" (this form needs Redis 6.2 or later). PUBLISH is fire and forget: a server that is not subscribed at that moment misses it. That is safe, because the message is already in the stream and the client syncs from there.

Example 2: Finding the recipient's server

With many servers, a message must reach the one that holds Ben's connection. A pub/sub channel or a session store tells the sender's server where to forward it.

Ana is connected to server 1 and Ben to server 2. Server 1 looks up where Ben is, then forwards the message to server 2, which pushes it to his device.

Example 3: A server fails

When a chat server dies, its connections drop. Clients reconnect to another server and ask for messages after their last ID. Nothing that was acknowledged is lost, because everything was saved first. A message the sender's app had not yet had acknowledged is kept by that app and retried with the same ID.

The client has messages 1 to 4. After reconnecting it asks for everything after 4 and receives 5 and 6, which were saved while it was disconnected.

More real-world scenarios

4. Group chats

A message to a group of a hundred people is stored once and delivered to each member's connections. Very large groups switch to pull for quiet members. For a group of 100 (illustrative), one message means one write and up to 99 deliveries, so the fan-out cost grows with the number of members, not the number of messages.

5. Offline users and push notifications

A user whose app is closed gets a push notification from the phone's push service, then fetches the messages when the app opens.

6. Read receipts

Each device sends an acknowledgement when a message is shown. The server stores the highest read ID per conversation instead of a flag per message.

7. Presence

An "online" dot comes from heartbeats with an expiry. Services avoid broadcasting every change by sending presence only to people looking at that contact.

8. Customer support chat

A web widget connects over WebSocket to an agent queue. Transcripts are stored for the customer and the agent.

9. Live comments on a stream

Viewers send comments that are fanned out to everyone watching. Rate limits and sampling stop the screen flooding.

10. Multi-device sync

One account has a phone, a laptop and a tablet. Each device keeps its own last-seen ID and syncs independently.

Tools and managed services

Kind	Name	Notes
Proxy	NGINX, HAProxy, Envoy	Hold and route WebSocket connections
Pub/sub and log	Redis, Apache Kafka	Fan messages out between servers and keep history
Storage	Apache Cassandra, PostgreSQL	Messages partitioned by conversation
Managed	AWS API Gateway WebSocket APIs	Hosted WebSocket connections
Mobile push	Apple Push Notification service, Firebase Cloud Messaging	Wake apps that are closed

The takeaway

Whenever users need to see each other's messages in real time, hold an open connection, save each message before delivering it, and let a reconnecting client catch up from its last ID.

Chat System

Key takeaways

Why a chat system is a classic design question, and what it teaches

Rough capacity estimation

Try it

Chat System

Ways to deliver messages

One chat server vs many

High-level design

How message delivery works

Chat design vs the alternatives

Avoiding a new single point of failure

Bottlenecks and how to relieve them

Common mistakes

How to talk about a chat system in an interview

Where you meet chat systems in real systems

Example 1: Accepting WebSocket connections and storing messages

Example 2: Finding the recipient's server

Example 3: A server fails

More real-world scenarios

4. Group chats

5. Offline users and push notifications

6. Read receipts

7. Presence

8. Customer support chat

9. Live comments on a stream

10. Multi-device sync

Tools and managed services

The takeaway

Interview questions