Latency does not grow in proportion to load. In the simplest queueing model the mean time in system is the service time divided by 1 − ρ: doubling utilization from 45% to 90% multiplies it by 5.5, and at ρ ≥ 1 there is no steady state at all. Real services are not that model, but they inherit its shape — a knee, then a wall — so headroom is decided where the curve bends on measured data, and the load test that measures it must be open-loop, or it hides the wall it was run to find.
| Utilization ρ | Mean jobs in system, ρ/(1−ρ) | Mean time in system, in service times, 1/(1−ρ) |
|---|---|---|
| 0.5 | 1 | 2 |
| 0.8 | 4 | 5 |
| 0.9 | 9 | 10 |
| 0.95 | 19 | 20 |
| 0.99 | 99 | 100 |
What is a theorem, and where. The table is M/M/1 in steady state: Poisson arrivals at rate λ, exponential service at rate μ, one server, an unbounded queue, and ρ = λ/μ < 1 — without that last condition the chain is not positive recurrent and the queue has no stationary length. Inside those assumptions the numbers are exact; outside them they are an analogy. Little’s law L = λW is different: it holds on any single observed sample path over a finite window, with no assumption about arrival or service distributions, stationarity or queue discipline. That makes it an engineering tool, not a model: requests in flight = arrival rate × mean latency, so 2,000 req/s at 50 ms mean needs about 100 concurrent slots (a pool, a connection limit, a semaphore), and a dashboard whose three numbers disagree is measuring something wrong. It relates means only; it says nothing about p99.
What is a heuristic. Any fixed “keep CPU under N%” rule. Where the knee sits depends on the number of servers, service-time variability, queue bounds and the averaging window — a 60-second mean of 60% can contain seconds at ρ > 1, and those seconds are where the tail lives (26). So the rule is procedural: measure latency against offered load (37), find the knee, set the operating point below it by the size of the bursts and failover load you must absorb, and decide in advance what happens past the wall — bounded queues and shedding (23), per-tenant admission (103). Retries raise λ exactly when ρ is highest, so a retry budget (24) is part of capacity, not of error handling. Throughput versus node count is a separate curve (38, 89).
The load generator decides what you see. A closed generator sends the next request only after the previous one returns; an open one sends on a schedule. Schroeder, Wierman and Harchol-Balter found that at equal load mean response times are markedly lower in the closed model, that a closed system approaches the open one only slowly as its population grows, and that most of the web workload generators they surveyed were closed. A closed test cannot push a server past its capacity: it slows down with the server and reports a comfortable latency. The same mechanism inside the measurement is Tene’s coordinated omission: a stall delays the requests that should have been sent during it, and timing from the actual send drops them. Pick the model from how your traffic arrives — independent users are open, a fixed worker pool is closed — and for open traffic measure from the intended send time (wrk2, or the correction in HdrHistogram).
import asyncio
import random
import time
async def closed_loop(send, workers, duration_s):
# The next request waits for the previous response: when the server
# stalls, each worker stalls with it, records one slow sample and sends
# nothing meanwhile, so the stall is under-counted (coordinated omission).
latencies = []
deadline = time.perf_counter() + duration_s
async def worker():
while time.perf_counter() < deadline:
started = time.perf_counter()
await send()
latencies.append(time.perf_counter() - started)
await asyncio.gather(*(worker() for _ in range(workers)))
return latencies
async def open_loop(send, rate_per_s, duration_s, rng=random):
# Arrivals follow the clock, not completions. Latency is measured from
# the intended send time, so a stall anywhere is charged to every request
# it delayed instead of being omitted. In-flight requests are not capped,
# as in an open system: at rho >= 1 latency grows for as long as the run.
latencies = []
start = time.perf_counter()
async def one(intended):
await send()
latencies.append(time.perf_counter() - intended)
tasks = []
intended = start
while intended < start + duration_s:
await asyncio.sleep(max(0.0, intended - time.perf_counter()))
tasks.append(asyncio.create_task(one(intended)))
intended += rng.expovariate(rate_per_s) # Poisson arrivals, as in M/M/1
await asyncio.gather(*tasks)
return latencies