Health Checks, Draining & Graceful Rollout

Send traffic only to nodes that can serve it

Level 1's load-balancing lesson named health checks as the mechanism that keeps a pool healthy; this lesson is the deep treatment. A load balancer only helps if it sends traffic to nodes that can actually serve it and stops sending to nodes that cannot. That is the job of health checks, and the subtlety is doing it without evicting healthy nodes or dropping in-flight work during a deploy.

There are two ways to know a node is bad. Active checks have the LB probe each backend on an interval (an HTTP GET /healthz, a TCP connect) and mark it unhealthy after N consecutive failures: fast, proactive detection at the cost of probe traffic. Passive checks (outlier detection) observe real traffic: if a backend starts returning 5xx or timing out on actual requests, eject it from the pool for a cooldown. Passive checks catch failures a shallow probe misses. Production uses both.

Liveness vs readiness

• Liveness asks "is this process alive at all?" A failed liveness check means the node is broken and should be restarted/replaced.
• Readiness asks "is this node ready to receive traffic right now?" A node can be alive but not ready: still warming its cache, loading a model, filling connection pools, or temporarily shedding load. A not-ready node should be pulled from the LB pool but not killed.

Conflating them is a classic bug. If you treat "not warmed up yet" as a liveness failure, the orchestrator keeps killing and restarting perfectly good nodes in a crash loop. Gate a newly started node behind readiness until it is warm, then admit it.

Deploys: draining and slow-start

• Connection draining (graceful shutdown): when a node is going away, first mark it not ready so the LB stops sending it new requests, but let its in-flight requests (and long-lived streams) finish up to a drain timeout before the process exits. The sequence: stop advertising -> stop new traffic -> wait for in-flight to complete (or hit the deadline) -> terminate.
• Slow-start / ramp: a freshly joined node starts with zero warm cache and cold connection pools. If the LB immediately gives it a full 1/N share, it can fall over or spike latency. Slow-start ramps its traffic share up over some seconds.

Interview nuance: deep vs shallow health checks. A shallow check returns 200 as long as the web server is up, even if the database or a critical downstream is unreachable, so the node keeps receiving traffic and failing every real request. A deep check verifies the critical dependencies. But deep checks have their own trap: if every node's health check hits a shared dependency and that dependency blips, every node marks itself unhealthy at once and the whole fleet drops out, turning a minor blip into a total outage. The mature answer: deep enough to catch a truly broken node, with hysteresis, and not so coupled that a shared-dependency blip fails the entire fleet simultaneously.

  drain sequence on node removal / deploy:
  mark NOT-READY -> LB stops NEW traffic -> in-flight finishes (<= drain deadline) -> terminate
  join sequence:
  start -> READINESS gates traffic until warm -> slow-start ramps share up

Recap: use active probes plus passive outlier ejection; keep liveness (restart) separate from readiness (pull from pool, do not kill); drain connections and slow-start new nodes so a rolling deploy drops nothing; and make checks deep enough to catch a broken downstream without letting one shared-dependency blip fail the whole fleet at once.