TheLearningLoopWhy you need to think about scale, and when to act
A system that works for 100 users can fall over at 100,000. The code is the same, but one machine runs out of CPU, memory or network, requests queue up, and users see slow pages and errors. Scalability is a system's ability to keep working well as the load grows.
Planning for it gives you two things interviewers always ask about:
- Capacity: handle more requests per second as demand grows, without a rewrite.
- Availability: keep serving when a machine fails, because there is more than one.
Think about scale when:
- Traffic is growing, or has peaks such as a sale, a launch or a live event.
- You have a target, such as "10 times today's traffic", or a latency promise to users.
- One machine is a single point of failure and downtime is costly.
Skip the heavy machinery, or keep it simple, when:
- You are small and one server is enough: measure first. Early scaling work costs time and money.
- The problem is a slow query or slow code: fix that before adding machines. Performance problems and scalability problems are different (see below).
- Traffic is small and steady: a bigger machine (scale up) is often the simplest fix.
Try it
Pick how the system reacts as traffic changes, then step through the 16 ticks. Watch the capacity line chase the load line, and what happens to latency.
Scaling up and scaling out
Step 1 of 17 · Press Space to play, arrow keys to step- Server 1
100 req/s
serving
Start: one server that handles 100 requests per second. Traffic will change over 16 ticks. Pick how the system reacts, then step through.
- load (req/s):
- 0
- capacity (req/s):
- 100
- servers:
- 1
1 tick = one step of the simulation. Load and capacity are requests per second. Latency is a simple model: it climbs steeply as a server nears full, and a request that cannot be served times out at 3000 ms.
Things to try:
- Steady growth with Do nothing: traffic outgrows the single server early and 1,440 requests per second are dropped in total. Switch to Scale out: the servers keep up, nothing is dropped, and the worst latency is 200 ms.
- Steady growth with Scale up: each upgrade needs a restart, so the service is down for a tick each time, and 300 requests per second are dropped. A bigger machine is not free of downtime.
- Tick the box Crash a server at tick 9. With Scale out nothing is dropped and the worst latency doubles to 400 ms. With Scale up a crash costs far more, because there was only one server: 740 are dropped in total.
- Sudden spike with Scale out: it still drops 450, because the new servers only arrive after the spike has hit. Autoscaling reacts, it does not predict.
Scaling strategies
| Strategy | How it works | Good when | Weakness |
|---|---|---|---|
| Scale up (vertical) | Move to a bigger machine: more CPU, memory or disk | Simple apps, databases that are hard to split, quick fixes | A ceiling (the biggest machine), often downtime to resize, one machine is still a single point of failure |
| Scale out (horizontal) | Add more machines and spread the load | Web and API servers, anything stateless | Needs a way to spread traffic and servers that do not keep user state; more moving parts |
| Reactive autoscaling | Add or remove machines automatically when a measurement such as CPU crosses a limit | Traffic that changes gradually | Reacts after load arrives, and new machines take time to start |
| Scheduled or pre-scaling | Add machines before a known peak, on a timetable or ahead of an event | Predictable peaks: sales, launches, daily rushes | Needs you to know when the peak comes; wasteful if you guess wrong |
Latency vs throughput, and performance vs scalability
- Latency is how long one request takes, for example 120 ms. Throughput is how many requests finish per second, for example 2,000. A system can have low latency and low throughput, or the reverse.
- Utilisation links them. When a server is quiet, latency is steady. As it nears full, requests queue, and latency climbs steeply. This is why systems are run well below 100% busy.
- Performance is how fast the system is for one user or one request. Scalability is whether that performance holds as load grows.
A system with a performance problem is slow even for a single user. A system with a scalability problem is fast for one user and slow under load. The fixes differ: optimise code and queries for the first, add capacity or spread work for the second.
How autoscaling decides
Autoscaling follows a loop, and every step has a delay:
- Measure a signal every so often, such as average CPU, requests per second or queue length.
- Compare it with a target, for example "keep average CPU near 70%".
- Decide how many machines are needed, within a minimum and a maximum you set.
- Start new machines. They take time to boot, load the app and pass health checks.
- Wait before scaling again, so the system does not flap up and down.
Because of the delays, the system always trails demand a little. That is why you keep headroom (run below the target), set a sensible minimum, and use pre-scaling for known peaks. The maximum protects your budget.
Scaling up vs scaling out vs the alternatives
| Concern | Scale out | Scale up | Optimise the code | Over-provision |
|---|---|---|---|---|
| Survives a machine failure | Yes | No | No | Only if there is more than one machine |
| Has a hard ceiling | No, in practice | Yes | Yes | Set by what you bought |
| Needs app changes | Usually (stateless servers) | No | Yes, by definition | No |
| Downtime to change size | None | Often a restart | None | None |
| Cost when quiet | Low with autoscaling | Pays for the big machine | Lowest | High |
| Best for | Web and API tiers | Quick fix, databases | Slow code and queries | Known, short, critical peaks |
Optimising is always worth doing first when something is slow for one user. Scaling out is the long-term answer for growth, and scaling up is the fastest answer for today.
Avoiding a new single point of failure
Scaling out spreads the app servers, but it moves the weak spots elsewhere:
- The load balancer in front of the servers must itself be redundant.
- Shared state such as sessions has to live outside the servers, in a shared store, or any server can no longer serve any user.
- The database is usually the next bottleneck and the next single point of failure. Plan for read replicas, caching and, later, sharding.
The trade-off: you gain capacity and resilience, and you accept more machines to run, more network calls and harder debugging.
Common mistakes
- Scaling the wrong tier. Adding web servers does nothing if the database is the bottleneck. Measure to find the slow part first.
- Keeping state on the server. Sessions or files stored on one machine stop it being replaceable. Move them to a shared store.
- Quoting averages only. Averages hide slow requests. Track the slowest few percent of requests (p95 or p99).
- Trusting autoscaling for sudden spikes. It reacts late. Pre-scale for known events and keep headroom.
- No maximum. A bug or an attack can scale you into a huge bill. Always set a limit and an alert.
- Running near 100% busy. Latency explodes close to full. Aim for comfortable headroom.
How to talk about scalability in an interview
- Start with numbers: users, requests per second, data size, and the latency target. Say which you are assuming.
- Prefer scale out for stateless tiers and say why: no ceiling, survives a failure.
- Say where the next bottleneck is, usually the database, and name the fix: cache, replicas, sharding.
- Mention failure and spikes unprompted: redundancy, health checks, headroom and pre-scaling.
- Name the trade-off: more machines to run in return for capacity and resilience.