Chaos engineering: failure hypotheses are tested by experiment

“We will survive losing a replica” is a hypothesis; until it has been tested by injecting the failure, it is an opinion. The method: steady-state metric → hypothesis (“killing a pod does not move p99”) → minimal blast radius (27) → injection → comparison. The detail people skip: the end goal is production, because staging reproduces neither the traffic, nor the data, nor the configuration.

The practice comes from Netflix: Chaos Monkey, which terminated production instances during business hours, appeared around its move to AWS, and the discipline was written up in Basiri et al., “Chaos Engineering” (IEEE Software, 2016) and in the Principles of Chaos Engineering. The principles are the method above: define steady state as a measurable output, hypothesize that it continues in both the control and the experimental group, vary real-world events, run in production, automate, minimize blast radius.

# an experiment is a versioned artifact, like a test
experiment: kafka-broker-loss
steady_state: { metric: consumer_lag_p99, threshold: "< 30s" }
method: [{ action: kill-pod, selector: "app=kafka, broker=2" }]
rollback: automatic          # abort on threshold breach
blast_radius: cell-eu-1      # one cell, not the fleet

An honest assessment. Chaos without observability is vandalism: if you cannot answer “what broke?”, you have no business breaking anything. The order is strict: SLOs and tracing first, injections second. Game days on staging are a legitimate first step, but not the final one.

What a good experiment finds. Not that the broker died — you knew that — but the assumption around it: a producer configured with acks=1 that loses acknowledged writes on leader failover, a client whose retry storm outlasts the outage (24), an alert that never fired. Each finding becomes a fix plus a permanent automated experiment, like a regression test.

Limits. Fault injection tests resilience, not correctness: a consistency bug under partition needs history checking against a model (98), not a dashboard. Where a failed experiment harms people or breaks a regulatory duty, production injection is replaced by simulation and restore drills (97). And for a small system with one region and an untested backup, the restore test pays off long before the monkey does.