Autoscaling & Elasticity

Four scalers, four problems

Elastic scaling, matching capacity to load automatically, is the core reason to run cloud-native. There are four distinct scalers and they solve different problems; naming them precisely is the interview signal.

• HPA (Horizontal Pod Autoscaler): adds and removes Pod replicas to hit a target metric. The workhorse for stateless services.
• VPA (Vertical Pod Autoscaler): right-sizes a Pod's CPU/memory requests. Useful for workloads that cannot scale horizontally, but it usually restarts the Pod to apply, so it does not fight HPA on the same metric.
• Cluster Autoscaler / Karpenter: adds and removes nodes when Pods cannot be scheduled (Pending) or when nodes are underused. HPA makes more Pods; the cluster autoscaler makes room for them.
• KEDA (Kubernetes Event-Driven Autoscaling): scales on external event sources (Kafka lag, SQS depth, Redis list length, cron) and, critically, can scale to zero when the source is empty.

Scale on the right signal

The most important senior point is that CPU is the default HPA metric and it is often wrong. For a web API, requests-per-second or p99 latency tracks user experience far better than CPU, which may sit low while the service is latency-bound on a downstream. For a queue consumer, the correct signal is queue depth or consumer lag: if 100,000 messages are backed up, you want to scale on that backlog directly, not on the CPU of the current workers (which may look fine while the backlog grows unbounded). Use custom or external metrics (via the metrics adapter or KEDA) and set a percentile target, for example keep p99 under 200 ms rather than average CPU at 70 percent.

Scale-to-zero and cold starts

Scaling to zero saves money on spiky, event-driven work, but the first request after zero pays a cold start: pull image, boot process, warm caches, which can be hundreds of ms to seconds. Mitigations: keep a small warm pool (a floor of 1 to 2 replicas so you never fully cold-start on the user path), use provisioned/pre-warmed concurrency, and shrink the image and boot path. The decision is explicit: pure scale-to-zero for a nightly batch or a rare webhook, a warm floor for anything a user waits on synchronously.

Diurnal and spiky patterns: for predictable daily cycles use scheduled or predictive scaling to pre-provision before the morning ramp so autoscaling is not racing the traffic wave. To avoid flapping (rapidly scaling up and down around the threshold), set stabilization windows and sensible scale-down delays so a brief dip does not tear down capacity you will need again in 30 seconds.

Interview nuance: if you say "scale on CPU" for an event-driven or latency-bound service, a strong interviewer will push: "what if CPU is at 40 percent but the queue has a million messages?" The correct answer scales on backlog or p99, and uses KEDA for the queue-depth and scale-to-zero case.

Recap: pick the scaler to the problem (HPA Pods, cluster autoscaler nodes, KEDA events with scale-to-zero), scale on the signal that reflects user pain (RPS, p99, queue depth) not reflexive CPU, and blunt cold starts with a warm floor and stabilization windows.