Measure the tail, not the mean: when one user request fans out to many servers, it waits for the slowest of them, so a rare slow response on each server becomes a common slow response for users. Hedged requests attack the tail directly: send a duplicate to another replica once the first has been outstanding longer than the p95, and take whichever answers first.
The source is Jeffrey Dean and Luiz André Barroso, “The Tail at Scale” (Communications of the ACM, February 2013). Their point is that at Google’s scale latency variability cannot be engineered away component by component — background jobs, garbage collection, queueing and shared resources always produce some slow responses — so systems have to be tail-tolerant, the way fault-tolerant systems accept that parts fail. Besides hedging they describe tied requests (both replicas are told about each other and the second cancels once the first starts work), micro-partitioning for fast rebalancing, and putting slow machines on probation.
// hedged request on virtual threads
try (var scope = StructuredTaskScope.open(Joiner.<Reply>anySuccessfulOrThrow())) {
scope.fork(() -> replica1.get(key));
scope.fork(() -> { Thread.sleep(p95Latency); return replica2.get(key); }); // hedge
return scope.join(); // first success wins, the other is cancelled
}
// On JDK 25 (JEP 505) the same joiner was named anySuccessfulResultOrThrow().
// On JDK 21 (JEP 453) the shape differs; the factories did not exist yet:
// try (var scope = new StructuredTaskScope.ShutdownOnSuccess<Reply>()) {
// scope.fork(() -> replica1.get(key));
// scope.fork(() -> { Thread.sleep(p95Latency); return replica2.get(key); });
// scope.join();
// return scope.result();
// }The arithmetic and the cost. With a fan-out of 100 independent calls, the probability of hitting at least one component's tail is ≈ 1 − 0.99¹⁰⁰ ≈ 63 %: the component's p99 becomes the user's ordinary experience. Hedging applies only to idempotent, cancellable operations (22) and buys latency with extra traffic, so it needs a budget, and the share of hedged requests belongs in the metrics.
What was measured and what was not. The 63 % figure is arithmetic under an independence assumption; real tails are partly correlated (one GC pause, one noisy neighbour hits many requests at once), so treat it as the shape of the effect, not a prediction. The measured part is the paper’s: in a Google benchmark reading 1,000 keys spread over 100 BigTable servers, hedging after a 10 ms delay cut the 99.9th-percentile latency from 1,800 ms to 74 ms while sending about 2 % more requests. Hedging does not help when the slowness is shared — an overloaded database behind every replica only gets twice the load — and it is the wrong tool for non-idempotent writes.