Measure, then optimize

heuristic
established
fragment

Versions. testing.B.Loop — Go 1.24+.

Conflicts with: “Network is the scarce resource” (MapReduce) ↔ “network is rarely the bottleneck” (Ousterhout et al.)

References: More predictable benchmarking with testing.B.Loop, net/http/pprof

Knuth’s famous line is a claim about measurement, not a ban on optimization: guesses about where a program spends its time are usually wrong, so the critical part is found with a profiler first and optimized second.

The source is Donald Knuth, “Structured Programming with go to Statements” (ACM Computing Surveys, 1974). The full passage: “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.” And right after it he argues that a priori judgments about which parts are critical are often a mistake, because programmers who use measurement tools find that their intuitive guesses fail. The quote is usually cut before the second sentence, which turns a rule about where to look into an excuse not to look.

What it looks like in a backend. A nightly job takes 40 minutes and the team suspects the JSON parser. A profile shows 70 % of wall time waiting on the database: one query per row, an N+1 that EXPLAIN ANALYZE and a batched read fix in an afternoon. The parser was 4 %; by Amdahl (89) making it infinitely fast would have saved under two minutes.

fragment

Java · JMH

// the only correct way to microbenchmark on the JVM
@Benchmark
@BenchmarkMode(Mode.SampleTime)
public Portfolio settle(BenchState s) { return Settlement.settle(s.trades, s.rates, DAY); }

// not System.nanoTime around a loop: JIT, warmup, DCE turn that into a lottery

Rust · criterion

// statistics + DCE protection via black_box
c.bench_function("settle", |b| b.iter(|| settle(black_box(&trades), &rates, day)));

// built-in harness + profiles from production (net/http/pprof — always on)
func BenchmarkSettle(b *testing.B) {
    for b.Loop() { Settle(trades, rates, day) }
}

// go tool pprof -http=: http://svc:6060/debug/pprof/profile

Python

# pyperf for micro; py-spy/scalene for prod: sampling, no instrumentation
$ py-spy record --pid 4711 --format speedscope

Caveats to the examples. JMH is the standard for JVM microbenchmarks, but not the only correct way to measure: production profiles and load-test rigs answer different questions. testing.B.Loop requires Go 1.24+. net/http/pprof is not “always on”: the package has to be imported and an HTTP handler exposed, and in production it must be locked down by network and authentication — it is a debugging surface.

Where “measure first” is not enough. Some decisions cannot be measured after the fact because they are expensive to undo: data layout (10, 39), the partition key (92), a synchronous call in the request path. Those belong to Knuth’s critical 3 % and are measured with a prototype before they are built. And a measurement can lie: a microbenchmark that the JIT optimized away, a load test that cannot exceed capacity (100), a profile from a dataset ten times smaller than production.