Deterministic replay, dry-run modes, inspectable intermediate state. Without them the time to find a production bug is bounded only by luck: the failure cannot be reproduced, the change cannot be previewed, and the state that led to it is gone.
The strongest form of the idea is deterministic simulation testing, popularized by FoundationDB: Will Wilson’s talk “Testing Distributed Systems w/ Deterministic Simulation” (Strange Loop 2014) describes running the whole cluster in one thread with simulated network, disk and clock, so that a seed reproduces a failure exactly. TigerBeetle’s simulator follows the same design. The price is architectural, which is why the principle is a requirement rather than a tool: every source of nondeterminism — time, randomness, threads, I/O — has to sit behind an interface from the start; retrofitting it into a codebase that calls the clock and the network directly is close to a rewrite.
// FoundationDB/TigerBeetle style: all I/O behind a trait,
// in tests, a deterministic simulator with a seed
trait Clock { fn now(&self) -> Instant; }
trait Net { fn send(&self, to: NodeId, msg: Msg); }
// prod: the real ones; test: SimClock + SimNet with controlled faults
// a test fails with seed=42 → reproduces byte for byte, every time# Dry-run as a mode, not as a flag checked in three ifs
class Executor(Protocol):
def apply(self, change: Change) -> None: ...
class RealExecutor: ...
class DryRunExecutor: # prints the plan; the logic ABOVE can't tell the difference
def apply(self, change: Change) -> None: print(f"WOULD apply: {change}")A dry run must build the same immutable plan that is then applied. Separate dryRun() and apply() branches drift apart, and they drift silently, exactly when someone is relying on the dry run.
The ordinary version is a plan file. terraform plan -out=plan writes the exact change set, and terraform apply plan applies that file and nothing else (86). The same shape works for a data backfill: compute the list of rows to change, persist it with a hash, let someone read it, then apply by hash. Replay needs the inputs, and that collides with privacy (36): recorded production traffic is personal data with a retention period, so record what reproduces the decision, not everything that passed by.