TLS Termination & Connection Management

Where do you decrypt, and how many connections do you hold?

Two questions decide the health of a system's front door under real load: where do you decrypt, and how many TCP connections does your backend actually have to hold open? Getting either wrong shows up as CPU burn, port exhaustion, or a load balancer that silently stops balancing.

Where you terminate TLS

The TLS handshake is expensive, and it happens per new connection. Edge/LB termination decrypts at the load balancer or edge PoP, so the backend fleet never pays the handshake cost and certificates are managed in one place. The gap: traffic from the LB to the backend is now plaintext internally. End-to-end TLS (or re-encryption at the LB) keeps it encrypted all the way, and mTLS additionally has both sides present certificates so services mutually authenticate: the zero-trust default inside a service mesh (Istio/Linkerd issue and rotate certs via sidecars). A common production shape: TLS terminated at the edge for the public handshake, then re-encrypted with mTLS for the internal hop, giving both cheap edge offload and a zero-trust interior.

SNI and certificates at scale. One IP/LB often fronts many hostnames. SNI (the client sends the target hostname in the ClientHello) lets the LB pick the right certificate per connection. Certificate rotation must be automated (ACME/Let's Encrypt, an internal CA); manual cert management does not survive thousands of short-lived certs.

Connection management: where the sharp edges live

Every new backend connection means a handshake and consumes an ephemeral port. A proxy opening a fresh connection per request will exhaust its ~64K ephemeral ports and burn CPU on handshakes. The fixes are keep-alive (reuse a connection for many requests) and connection pooling (a warm pool per backend, multiplexing requests over it), making backend connection counts a function of concurrency and pool size, not raw request rate.

The trap: multiplexed connections pin to one backend

HTTP/2 and gRPC multiplex many streams over one long-lived connection. A layer-7 load balancer balances at connection establishment. But a gRPC client opens one connection and keeps it, sending thousands of RPCs as streams over that single connection, all pinned to whatever backend the LB picked at connect time. Add ten new backend pods and existing clients keep hammering the old ones; the new pods sit idle.

gRPC client --- one long-lived H2 connection ---> backend A  (all streams pinned)
new backends B,C come up  ->  get zero traffic until clients reconnect

The fixes: client-side load balancing (the client spreads RPCs itself, via gRPC's round_robin resolver or an xDS/mesh control plane), balance at L7 per-request with a proxy that understands H2 streams (Envoy balances individual streams, not just connections), or periodically cycle connections (max-connection-age) so clients re-resolve and rebalance. WebSockets have the same pinning problem, so plan for connection draining and rebalancing on scale events.

C10k / C10M. Holding 100K-plus concurrent connections needs event-driven proxies (epoll/kqueue, nginx/Envoy) rather than thread-per-connection, plus OS tuning (file-descriptor limits, ephemeral port range, TCP buffers, SO_REUSEPORT).

Interview nuance: if you say "terminate gRPC at our L7 load balancer" without mentioning stream pinning, expect "then why did your new pods get no traffic after a scale-up?" Naming client-side LB or Envoy per-request balancing is what proves you have run this in production.

Recap: terminate TLS at the edge and re-encrypt with mTLS for a zero-trust interior, pool and keep-alive connections to avoid handshake cost and port exhaustion, and remember that long-lived multiplexed H2/gRPC/WebSocket connections pin to one backend so you need client-side or per-request balancing to actually spread load.