Server Concurrency Models: Thread-per-Request vs Event Loop & C10k

One question decides it: CPU-bound or IO-bound?

How a server maps incoming connections onto CPU work is one of the oldest and most consequential design choices, and it is entirely governed by one question: is the workload CPU-bound or IO-bound?

Thread-per-request

Classic Apache prefork, Tomcat's default, most synchronous frameworks: a thread (or process) per in-flight request. Its great virtue is simplicity: you write straight-line blocking code (result = db.query(...)), the OS scheduler handles switching, and stack traces are clean. The cost is that each thread carries a real price. A default Linux thread reserves around 1MB of stack, so 10,000 threads is roughly 10GB of address space before doing any work, and the scheduler pays context-switch overhead that grows with thread count. The killer is blocking IO: when a thread waits on a slow database or a downstream API, it is parked doing nothing but still consuming a thread. If your workload is 90% waiting on IO, your threads sit idle while CPU is nearly free, and you exhaust the thread pool (and memory) long before the CPU saturates. That is why a thread-per-request box can fall over at a few thousand concurrent connections while showing 10% CPU.

Event loop

Node.js, Nginx, Netty, Redis, Python asyncio (Go's runtime is a hybrid): one thread (or one per core) multiplexes thousands of sockets using an OS readiness API, epoll on Linux or kqueue on BSD/macOS, which lets the kernel say "these 50 of your 10,000 sockets have data ready" in one cheap call. Idle connections cost only a file descriptor and a little kernel memory, not a thread, so a single event-loop process holds hundreds of thousands of mostly-idle connections. This is precisely what IO-bound fan-out needs: an API gateway waiting on 20 backends per request spends almost all its time waiting, and the event loop turns that waiting into near-free multiplexing.

But the event loop has one absolute rule: never block the loop. Because one thread drives everything, any single long operation (a synchronous CPU task, a blocking file read, a JSON.parse of a 50MB payload) freezes every other connection until it finishes. A CPU-heavy image transcode on the event loop serializes the whole server behind it. The fix is to offload CPU work to a worker pool sized to the number of cores, keeping the loop free to do IO.

Interview nuance: The crisp rule is "event loops are for waiting, thread/worker pools are for computing." CPU-bound work does not benefit from an event loop because there is nothing to wait on; you are limited by cores, so you want exactly one busy worker per core, not async.

C10k and the OS limits

The C10k / C10M problem names the challenge of holding 10,000 (or 10 million) concurrent connections. It is unsolvable with one blocking thread per connection and requires non-blocking IO plus tuned OS limits:

• File descriptors: every socket is an fd, and the default ulimit -n is often 1024. Raise nofile (and system-wide fs.file-max) to hundreds of thousands.
• Ephemeral ports: a single source IP connecting to one destination IP:port is limited to roughly 28,000 outbound connections, so a proxy fanning out to one backend runs out of ports. Fix with connection pooling and spreading across multiple destination IPs/ports.
• Memory per thread: the ~1MB stack per thread that caps thread-per-request; event loops sidestep it by not having a thread per connection.

Thread-per-request:   [req]->thread->BLOCK on IO (idle, 1MB)   ... caps at ~thousands
Event loop:           epoll -> 1 thread -> 100k idle sockets   ... never block it
                                     `-> CPU task? offload to worker pool (N=cores)

Recap: CPU-bound work wants a worker pool sized to cores, IO-bound fan-out wants an event loop multiplexing many connections via epoll/kqueue, never block the loop with CPU or blocking IO, and past ~10k connections you must raise fd limits, pool connections around the ephemeral-port ceiling, and avoid the per-thread memory wall.