Load-Balancing Algorithms & Session Affinity

The rule for who gets the next request

Level 1 introduced load balancing at the L4/L7 layer and health checks; this lesson goes deep on the piece it left open, the algorithm that picks which backend serves the next request. Once a load balancer has a pool of healthy backends, it needs a rule for which one gets the next request. The rule matters because the wrong one creates hotspots: some nodes melt while others sit idle.

• Round robin (RR): hand requests out in rotation. Weighted RR biases toward bigger nodes. RR is perfect when every node is identical and every request costs about the same. It is blind to how busy a node actually is.
• Least-connections (least-outstanding-requests): send the next request to the node with the fewest in-flight requests. The right default when request durations vary widely. In a fleet where most requests take 5ms but some take 2s, a node that caught several 2s requests keeps getting new work on its RR turn even though it is buried; least-connections routes around it automatically because its in-flight count is high. This is the single most important algorithm intuition.
• Power-of-two-choices (P2C): for a large pool, tracking exact least-connections means an O(N) scan or a globally synchronized structure, and when many LB nodes independently pick "the least-loaded node," they herd onto whatever looked idle a moment ago. P2C fixes both: pick two backends at random, send to the less-loaded of the two. Nearly as good as true least-connections, O(1), no global state, provably avoids herding. The practical default for big fleets (Envoy, Nginx).
• Consistent / rendezvous hashing: hash a request key (user ID, session ID, cache key) to a backend so the same key always lands on the same node, and when a node joins or leaves only ~1/N of keys move. This is sticky routing without a lookup table: essential for cache-warm nodes and sharded in-memory state.

Session affinity: a targeted tool, not a default

Sticky sessions deliberately pin a client to one backend, usually via a cookie the LB sets or a hash of the client IP / session ID. You want it when a node holds warm per-user state and re-hitting the same node avoids a cold miss. But affinity has two real costs. First, uneven load: pinning means the balancer can no longer freely spread traffic, so a few heavy users skew load onto their pinned nodes. Second, lost state on node death: when the pinned node dies, everything it held is gone, and those users reconnect cold.

Interview nuance: the crisp story: "round robin for homogeneous stateless nodes; least-connections when request durations vary; power-of-two-choices when the pool is large; consistent hashing when I need stickiness or sharded state, accepting that stickiness costs even load and loses state on node death." If you reach for sticky sessions to compensate for not externalizing state, that is a design smell; if you reach for consistent hashing to keep a cache warm, that is sound engineering. Same mechanism, different justification.

   variable durations:
   RR  -> buried node still gets its turn   (bad: hotspot)
   LC  -> skip the buried node              (good)
   P2C -> pick 2 random, send to lighter    (good + O(1), no herd, no global state)

Recap: round robin for identical stateless nodes, least-connections for variable durations, power-of-two-choices as the O(1) no-herd default for large pools, and consistent/rendezvous hashing for sticky routing or sharded state; session affinity keeps a user on a cache-warm node but costs even load and loses that node's state when it dies, so use it deliberately.