Why does leader election require a majority of votes, and how does that prevent split brain?

More than half of five is three, and two disjoint groups cannot both contain three of five nodes. So during a network split at most one side can elect a leader, and the other side cannot. The trade-off is that the smaller side stops accepting writes, so you give up availability on that side to keep the data consistent.

Why do coordination clusters use three or five nodes and not four?

A majority of four is three, so four nodes tolerate one failure, the same as three nodes. Five tolerate two. The extra even node adds cost and a larger majority to reach without adding fault tolerance. The trade-off is that each extra node makes writes a little slower.

How would you build a distributed lock that is safe if the lock holder crashes or pauses?

Keep the lock in a consensus store such as etcd or ZooKeeper and tie it to a lease or an ephemeral node so it is released if the holder dies. Have each grant return an increasing fencing token and make the protected resource reject any request with an older token, so a paused holder that wakes up cannot do harm. The trade-off is that the resource has to check the token.

The leader of your service crashes. Walk through what happens before a new leader serves traffic.

Followers stop receiving heartbeats and one election timer runs out first. That node becomes a candidate in a higher term, a majority votes for it, and it becomes leader and starts sending heartbeats. Writes pause during this time, usually for a short while. The trade-off is a brief loss of write availability in exchange for never having two leaders.

Design a job scheduler that runs on several machines but must never run a job twice at the same time.

Run several scheduler copies and elect one leader through etcd or ZooKeeper using a lease. Only the leader schedules jobs, and a fencing token goes with each job so a stale leader is rejected by the workers. Make the jobs idempotent as a backstop. The trade-off is that failover takes a lease timeout, during which no new jobs start.

Consensus and Coordination | TheLearningLoop

Why you need consensus, and when to use it

Some questions need one answer that every machine agrees on, even when machines crash and the network drops messages: who is the leader, who holds this lock, what is the current configuration. If two machines each believe they are the leader, both accept writes and the data diverges. This is called split brain.

Consensus is the family of algorithms (Paxos, Raft and others) that let a group of nodes agree on a value so long as a majority can still talk to each other. Coordination is what you build on top: leader election, distributed locks, configuration and service discovery. It gives you two things interviewers always ask about:

Consistency: at most one leader or lock holder at a time, and every node sees the same decisions in the same order.
Availability through failover: when the leader dies, the survivors elect a new one without a human.

Use it when:

Exactly one node must do a job: run the scheduler, own a partition, be the database primary.
Several nodes must agree on shared state, such as cluster membership or configuration.
You need a lock across machines that is released automatically if the holder dies.

Skip it, or use something else, when:

A single database row or a SELECT ... FOR UPDATE is enough to guard the job.
Two nodes doing the same work is harmless, because the work is idempotent.
You would run your own cluster for one small lock. Use the coordination service your platform already has.
The data can be repaired later, so eventual consistency is acceptable.

Try it

Run a five-node cluster, then break it. The rule decides who is allowed to be leader.

Leader election

Step 1 of 3 · Press Space to play, arrow keys to step

RuleFailure

Initial state. Five nodes, N1 is the leader of term 1. Followers reset a timer on every heartbeat.

tick:: 0
leaders:: 1
highest term:: 1
writes committed:: 0
conflicting writes:: 0
messages:: 0

1 tick = one step. A write is committed only when a majority (3 of 5) confirm it.

Speed

Things to try:

Run the Raft-style rule with no failure. The leader confirms a write with a majority and commits it.
Make the leader crash. A follower's timer runs out first, it asks for votes, and a majority makes it leader of a higher term.
Choose Network split with the Raft-style rule, then with the naive rule. Only the three-node side can elect a leader in the first case. In the second, both sides have a leader and the writes conflict.
Watch the leaders counter. With the Raft-style rule it briefly shows 2 during the split, because the old leader N1 does not yet know it was replaced. That is safe: N1's write stays pending (it cannot reach 3 nodes) and is discarded once N1 sees the higher term. The conflicting writes counter stays at 0. With the naive rule both leaders commit, and that counter rises. The messages counter is the number of vote, heartbeat and confirmation messages sent so far.

Consensus building blocks

Building block	How it works	Good when	Weakness
Leader election	Nodes vote, and a candidate needs a majority. Each leader has a numbered term	One node must coordinate writes	Writes pause during an election
Replicated log (Raft, Paxos)	The leader appends entries and commits them once a majority store them	Replicating state in the same order everywhere	Every write waits for a majority round trip
Leases	A node holds a role for a time limit and must renew it	Locks and leadership that must expire if the holder dies	Clocks and pauses can let two holders overlap briefly
Fencing tokens	Each lock grant carries an increasing number, and the resource rejects older ones	Protecting a resource from a lock holder that was paused	The resource must check the token
Coordination service (etcd, ZooKeeper)	A small consensus cluster that other systems use for locks, election and configuration	You want these features without writing consensus yourself	Another cluster to run and keep healthy

Majority rule vs naive failover

Majority rule: a node can lead only if more than half the cluster agrees. Two leaders in the same term would need two disjoint majorities, which cannot exist. An old leader on the minority side may still believe it leads for a while, but it cannot get a majority to confirm a write, so nothing it accepts is ever committed. The price is that the smaller side of a split stops accepting writes, which gives up availability for consistency.
Naive failover ("I heard nothing, so I am the leader") stays available on both sides of a split, and then produces two leaders. It is fine only when you can merge or discard one side's work.

This is the trade-off in the CAP theorem. When the network splits, consensus systems choose consistency over availability on the minority side.

How a leader election works

A short version of Raft. The majority has a name, a quorum: the smallest group that is allowed to decide (3 of 5):

Every follower has a random election timer. A heartbeat from the leader resets it.
If the timer runs out, the follower becomes a candidate, adds one to the term, votes for itself and asks the others for votes.
Each node votes once per term. A candidate with a majority becomes leader.
If votes split, nobody wins, the timers run out again, and the random delays make a clean result likely next time.
A node that sees a higher term steps down at once. That is how an old leader, back after a split, learns it is no longer leader.
A node only votes for a candidate whose log is at least as up to date as its own, so a new leader always holds every entry that was already committed.

Numbers to keep in mind (illustrative defaults, all configurable): the Raft paper suggests election timeouts of 150 to 300 ms, and etcd's defaults are a 100 ms heartbeat and a 1000 ms election timeout. The timeout must be several times the network round trip, or healthy leaders get replaced for no reason. Each committed write costs at least one round trip to a majority, so cross-region clusters pay that latency on every write. Reads served by a leader that has not checked it is still the leader can be stale, so systems that need fresh reads confirm leadership first.

The leader then confirms every write with a majority before it counts as committed.

Coordination services vs the alternatives

Concern	etcd / ZooKeeper	A database row lock	Redis lock
Survives a node failure without losing the lock	Yes	Only with a replicated database	Not guaranteed
Lock is freed if the holder dies	Yes, with a lease or ephemeral node	With a timeout you build	With an expiry
Safe against a paused holder	With fencing tokens	With a version column	Not by itself
Extra system to run	A small cluster	None	A cache you may already run
Best for	Leadership, configuration, strict locks	Simple jobs with one database	Efficiency locks where a rare double run is harmless

A coordination service wins when correctness depends on one holder. A database row or a Redis lock is enough when a duplicate run is harmless.

Avoiding a new single point of failure

The coordination cluster is now critical for everyone who uses it. Run three or five nodes, never two or four: an odd number gives the same fault tolerance as the next even number with one machine fewer (three tolerate one failure, five tolerate two). Spread them across failure zones, keep the data small, and make clients cope with it being briefly unreachable. You accept slower writes, since each one needs a majority, and an operational burden.

Common mistakes

Running two or four nodes. You pay for more machines and tolerate no more failures than with one fewer.
Locks without expiry. A crashed holder blocks everyone for ever. Use a lease or an ephemeral node.
Trusting a lock after a pause. A garbage collection pause can outlast the lease, so use a fencing token.
Putting big or busy data in the coordination service. It is for small, important facts, not for your application data.
Checking the time to decide who is leader. Clocks drift. Timers are fine for noticing that a leader may have failed, but who is allowed to lead must come from terms and majorities, and who is allowed to write must be backed by fencing tokens.
Ignoring the minority side. Decide what a node does when it cannot reach a majority: stop writing.

How to talk about consensus in an interview

Say what you need: one leader, or one lock holder, at a time.
Say you would use etcd or ZooKeeper rather than write consensus yourself, and explain that a majority is why a split cannot create two leaders.
Mention leases that expire, terms, and fencing tokens unprompted.
Name the trade-off: consistency over availability, so the minority side stops and writes cost a majority round trip.

Where you meet consensus in real systems

Whenever a cluster must pick one leader or share one fact, a consensus cluster is underneath. Here are three drawn out, the first with commands you can run, then more examples.

Example 1: One leader for a background job

Three copies of a service run for availability, but only one should run the nightly job. Each copy tries to claim leadership in a coordination service. The claim is tied to a session or lease, so if the leader dies the claim disappears and another copy takes over. Open the example below to see the picture and the commands.

Config: leader election and a lockSetup a few commandsCode no code changes

Each copy tries to create the leader key. Only one succeeds, and the key is tied to a lease. If the leader stops renewing the lease, the key expires and another copy wins.

# etcd: a small consensus cluster. Commands use etcdctl.

# 1. A lease is a timer. If it is not renewed within 10 seconds, everything attached to it is deleted.
etcdctl lease grant 10
# prints: lease <lease-id> granted with TTL(10s)

# 2. Claim leadership only if nobody has it yet. The transaction is atomic.
#    Type each block, ending each one with an empty line. Replace <lease-id> with the id above.
etcdctl txn --interactive
#   compares:
#   create("/service/leader") = "0"
#
#   success requests (get, put, del):
#   put /service/leader node-1 --lease=<lease-id>
#
#   failure requests (get, put, del):
#   get /service/leader
#
# The first node prints SUCCESS and is the leader. Every other node prints FAILURE and sees the current leader.

# 3. The leader renews the lease while it is healthy. If it crashes, the key is deleted after the lease expires.
etcdctl lease keep-alive <lease-id>

# A distributed lock: runs the command while holding the lock "nightly-job", then releases it.
# Other callers wait. The lock is tied to a session, so a crashed holder is released automatically.
etcdctl lock nightly-job echo "running the job"

# A ready-made election: this blocks while node-1 is the leader.
etcdctl elect payments node-1

The etcd commands were run against a real etcd 3.5 server: granting a lease, claiming leadership with a transaction (the second caller got FAILURE and saw the first leader), lease expiry, lock with a command, and elect. The ZooKeeper commands were run against a real ZooKeeper 3.9 server: sequential ephemeral nodes, and a second create of /leader failing with "Node already exists". Names and times are examples.

Example 2: Choosing a database primary

When the primary database fails, the standby nodes must agree on exactly one new primary, or two would accept writes and diverge. A consensus service or a built-in consensus protocol makes that choice. Agreeing on one primary does not by itself guarantee it has every write the old primary acknowledged: with asynchronous replication, the last few writes may be missing, so the usual rule is to promote the most up-to-date standby and decide whether losing a few writes is acceptable.

The old primary has failed. The standbys and the coordination service agree on one new primary, and the other standby starts following it.

Example 3: A network split

If the network divides five nodes into two and three, only the group of three holds a majority. It keeps working. The group of two stops accepting writes instead of creating a second leader.

The group of three has a majority and elects a leader. The group of two cannot reach three votes, so it cannot elect one and refuses writes until the network heals.

More real-world scenarios

These scenarios describe common patterns in general terms. They are illustrative, not claims about how any one company runs its systems.

4. Kubernetes control plane

Cluster state is kept in etcd, and several controller copies use leader election so only one acts at a time.

5. Message brokers

A broker needs agreement on which node leads each partition, and moves leadership if a node fails. Some brokers have used an external coordination service for this. Kafka did so with ZooKeeper, and newer versions use KRaft, a Raft quorum built into the brokers themselves (ZooKeeper mode has been removed in recent releases).

6. Service discovery

Services register themselves under ephemeral keys, so a crashed service disappears from the list without cleanup code.

7. Distributed locks for scheduled jobs

A cron-style job runs on many machines, and a lock makes sure one machine runs it. A lease frees the lock if that machine dies.

8. Feature flags and configuration

A small shared store holds configuration. Services watch it and react when it changes, so every node sees the same value.

9. Distributed databases

Some databases replicate every write through a consensus protocol, so a majority holds each committed row and a lost node does not lose data.

10. Cloud control planes

Provisioning systems use leader election so that two controllers do not create the same virtual machine twice.

Tools and managed services

Kind	Name	Notes
Coordination service	etcd	Raft-based key-value store with leases, locks and watches
Coordination service	Apache ZooKeeper	Ephemeral and sequential nodes, used for years by Kafka (before KRaft) and Hadoop
Coordination service	HashiCorp Consul	Service discovery, health checks and a key-value store
Library	Raft implementations	Available for most languages if you must embed consensus
Kubernetes	Lease objects	Built-in leader election for controllers
Managed	Cloud-managed Kubernetes	Run the etcd control plane for you

The takeaway

Whenever exactly one node must lead or hold a lock, the first box you draw is a small majority-based coordination cluster such as etcd or ZooKeeper, with leases so a dead holder lets go.

Consensus and Coordination

Key takeaways