Load balancers

Someone has to direct the crowd

Last lesson you opened a second shop. Immediately, a question appears that never existed with one shop: when a customer walks down the street, WHO decides which shop they enter?

You hire someone to stand at the corner. Every customer talks to them first, and gets pointed to a shop: 'you - shop A, you - shop B.' On the internet, that person is called a load balancer. Every big website has one, and every request you have ever sent to Google or Instagram went through one first.

The balancer gives you a superpower that's easy to miss: customers only ever know about the person at the corner - never the shops themselves. So you can add shops, close shops, repaint shops... and no customer ever needs to know. The corner person just quietly starts (or stops) sending people there.

How does it choose? Three strategies

Strategy 1 - Take turns (round-robin). First customer to shop A, second to B, third to C, then back to A. Dead simple, totally fair, and it's the default everywhere. It works great when every customer wants roughly the same thing.

Strategy 2 - Least busy first. Look at the shops, count the customers inside each, send the newcomer to the emptiest one. This shines when orders are wildly different sizes - one customer buys a stamp (2 seconds), another orders a wedding cake (2 hours). With take-turns, one unlucky shop collects all the wedding cakes. Least-busy notices and self-corrects.

Strategy 3 - Same customer, same shop (sticky). Remember each customer's face and always send them to the same shop. When is this needed? Only when the workers are NOT forgetful - when a shop keeps something about you in its head. Notice something? If you followed last lesson's rule (forgetful workers, shared notebook), you never need this. Needing stickiness is usually a smell that state is hiding where it shouldn't be.

Your turn: be the balancer

Time to do the job yourself. In the widget below, YOU are the person at the corner. Send some requests with 'take turns' and watch them spread evenly. Then switch to 'least busy' and watch it favor idle servers. Finally - the important part - click a server to break it, and keep sending. Notice you never send anyone to the broken shop. How did you know it was broken? That's the next section.

Health checks: how the corner knows a shop is on fire

In the widget, you could SEE the broken server. A real load balancer can't see - so every few seconds, it walks over and knocks on each shop's door: 'you alive?' That knock is called a health check.

A good health check doesn't just ask 'is anyone there?' - it asks 'can you actually serve customers?' A shop can have its lights on while the till is broken. In computer terms: the health endpoint should check that the server can reach its database, not just that it's switched on.

Here's the math every engineer should do once. Say the balancer knocks every 5 seconds, and pulls a shop out after 3 missed knocks in a row. Then a shop that catches fire keeps receiving customers for up to 15 seconds - and every customer sent there during that window has a bad day. Knock more often and you find fires faster, but you also risk pulling out a shop that just blinked. It's a dial, not a right answer.

The polite goodbye: connection draining

One more trick separates amateur setups from professional ones. Sometimes YOU want to remove a shop on purpose - to renovate it (deploy new code). The clumsy way: lock the doors instantly. Every customer mid-order gets thrown out. They notice. They complain.

The polite way: the corner person stops sending NEW customers to that shop, but everyone already inside gets to finish their order. Once the shop is empty, it closes for renovation. Nobody was thrown out; nobody noticed anything.

This is called connection draining, and it's why the best websites can update their code twenty times a day without you ever seeing an error. When a deploy is invisible, this is the trick making it invisible.

Wait - who watches the watcher?

Sharp readers spot the loose thread: the person at the corner is now a single point of failure! If THEY faint, nobody gets directed anywhere, and all three healthy shops sit empty.

The industry's answer is refreshingly unclever: have two corner people, where the second one takes over the moment the first faints. And in practice most teams don't even manage that themselves - every cloud provider rents you a load balancer that is already a redundant team behind the scenes. In an interview, one sentence covers it: 'the balancer itself is redundant, typically managed by the cloud.' Then move on to the interesting parts.

And that's module 1. You now know: the four parts inside a server, the two ways to scale, the golden rule of statelessness, and the machine that makes many servers look like one website. Take the quiz below - module 2 tackles the shared notebook itself: databases, and what happens when THEY get crowded.