Software Engineering Principles: from SOLID to the AI Era

A “reason to change” is people, not requirements. Martin's later formulation: a module is responsible to one, and only one, actor — a group of people who originate requirements. A class that stores a shopping list and computes its cost violates SRP not because it “does two things”, but because different people own the list's contents and the pricing rules, and their requirements change independently. The actor is a criterion for where to cut, not a design driver: you don't create one class per stakeholder; you check the converse — that no single artifact serves two masters with mismatched change schedules.

None of this makes it operationalizable: granularity is arbitrary, the principle gives no stopping criterion for decomposition — a violation is recognized after the fact, and no design can be derived from it. The practical test is file history: commits from unrelated tasks by different people systematically converging on one class. The guard against the opposite extreme, splitting for the sake of splitting, is cohesion (13): what changes together lives together. The operationalizable replacement is information hiding (12): the question is not “how many responsibilities are here” but “what breaks outside if this internal decision is revisited”.

A contract has two layers. Signatures — which methods, which types — are checked by the compiler. Behavior — what a method promises to do — is checked by nobody but reviews and tests. LSP requires holding both: a subtype that matches the signatures but breaks the promise breaks everyone who wrote code against the supertype. The canonical example: Rectangle has setWidth and setHeight; Square, inheriting from it, must keep its sides equal — its setWidth(5) quietly changes the height too, and code that relied on “set the width, the height stays untouched” breaks precisely on the square. The compiler is happy; the contract is broken. The cure in the examples below is not to fix the inheritance but to remove it: a closed set of variants with no mutating setWidth (3), plus contract tests run against every implementation.

An LSP violation and its fix: the only SOLID principle that deserves examples.

// sealed instead of inheritance — the contract is closed and compiler-checked
sealed interface Shape permits Rect, Square {}

record Rect(double w, double h) implements Shape {}
record Square(double side) implements Shape {}

// Square extends Rectangle is impossible by construction — no mutating setWidth

// LSP is expressed through trait contracts; a violation is a panic instead of Err
trait Storage {
    fn get(&self, k: &str) -> Result<Option<Vec<u8>>, StorageErr>; // contract: never panics
}

// an impl that calls unwrap() inside get violates LSP,
// caught by code review + proptest

// io.Reader's contract lives in its docs; an implementation that panics instead of
// returning an error violates LSP, even though the compiler is happy
type SafeReader struct{ inner io.Reader }

func (r SafeReader) Read(p []byte) (int, error) {
    n, err := r.inner.Read(p)
    if err != nil && err != io.EOF {
        return n, fmt.Errorf("read: %w", err) // not panic
    }
    return n, err
}

# Protocol + a contract test run against every implementation
from typing import Protocol

class Storage(Protocol):
    def get(self, key: str) -> bytes | None: ...

def storage_contract_suite(impl: Storage):   # one test suite for all implementations
    assert impl.get("missing") is None       # not KeyError — that's the contract

A function returns a type that cannot be invalid by construction. Validation throws knowledge away; parsing keeps it in the type.

The everyday equivalent is connectors. A power plug does not fit a USB port not because someone inspects it at the port, but because the shape is different. Validation is a “do not insert here” sticker; parsing is a differently shaped connector. The first kind of error is caught at runtime, and only if nobody forgot the check; the second is never made at all. The same analogy carries over to the neighbouring cards: 3 — there are exactly as many connectors as there are meaningful variants, and an extra one cannot even be manufactured; LSP (1) — if the plug fits, the pins must carry the voltage promised on the label.

// record + compact constructor = smart constructor
public record Email(String value) {
    public Email {
        if (!value.matches(".+@.+\\..+")) throw new IllegalArgumentException(value);
    }
    public static Optional<Email> parse(String raw) {
        try { return Optional.of(new Email(raw)); }
        catch (IllegalArgumentException e) { return Optional.empty(); }
    }
}
// downstream, Email is always valid — no repeated checks

// newtype with a private field — the canonical form
pub struct Email(String);          // private field, can't be built from outside

impl Email {
    pub fn parse(raw: &str) -> Result<Self, EmailError> {
        raw.contains('@').then(|| Email(raw.to_owned())).ok_or(EmailError::Invalid)
    }
    pub fn as_str(&self) -> &str { &self.0 }
}

// private field + constructor — the only way in
type Email struct{ value string }   // zero value is useless — on purpose

func ParseEmail(raw string) (Email, error) {
    if !strings.Contains(raw, "@") {
        return Email{}, fmt.Errorf("invalid email: %q", raw)
    }
    return Email{value: raw}, nil
}

func (e Email) String() string { return e.value }

# frozen dataclass + factory; NewType for cheap cases
from dataclasses import dataclass

@dataclass(frozen=True, slots=True)
class Email:
    value: str

    def __post_init__(self):
        if "@" not in self.value:
            raise ValueError(f"invalid email: {self.value!r}")

# or pydantic: class Email(BaseModel): value: EmailStr — same thing, on steroids

Sum types instead of flag combinations. {isLoading, data, error} has 4 impossible states out of 8; a sum type has zero.

Why eight: three independent fields, each either present or absent — 2³. Four of them make sense: nothing started, loading, data arrived, error arrived. The other four — “loading and data at once”, “data and error”, “loading and error”, “everything at once” — are representable in the type, unreachable in meaning, and every code branch has to decide what to do with them. Usually it doesn’t, and the impossible state reaches the user. A sum type removes the question itself: there are exactly four states, because a fifth cannot be expressed.

// sealed + record + pattern matching
sealed interface FetchState<T> permits Loading, Loaded, Failed {}
record Loading<T>() implements FetchState<T> {}
record Loaded<T>(T data) implements FetchState<T> {}
record Failed<T>(Exception cause) implements FetchState<T> {}

String render(FetchState<String> s) {
    return switch (s) {                     // compiler demands exhaustiveness
        case Loading<String> l -> "spinner";
        case Loaded<String>(var data) -> data;
        case Failed<String>(var e) -> "error: " + e.getMessage();
    };
}

// enum is native; exhaustiveness comes for free
enum FetchState<T> {
    Loading,
    Loaded(T),
    Failed(anyhow::Error),
}

// match without a wildcard: add a variant and it won't compile until handled

// no sum types; the closest substitute is a sealed interface + type switch.
// Honest assessment: exhaustiveness is checked by a linter, not the compiler
// (go-check-sumtype / exhaustive)
type fetchState interface{ isFetchState() }

type Loading struct{}
type Loaded struct{ Data []byte }
type Failed struct{ Err error }

func (Loading) isFetchState() {}
func (Loaded) isFetchState()  {}
func (Failed) isFetchState()  {}

# union + match; mypy/pyright check exhaustiveness via assert_never
from dataclasses import dataclass
from typing import assert_never

@dataclass(frozen=True)
class Loading: pass

@dataclass(frozen=True)
class Loaded: data: bytes

@dataclass(frozen=True)
class Failed: cause: Exception

type FetchState = Loading | Loaded | Failed

def render(s: FetchState) -> str:
    match s:
        case Loading():      return "spinner"
        case Loaded(data):   return data.decode()
        case Failed(cause):  return f"error: {cause}"
        case _:              assert_never(s)   # static exhaustiveness

An error is part of the signature, not a side channel for control flow. Exceptions break locality: any call can jump anywhere.

// checked exceptions: a historically failed attempt at the same thing;
// the modern form is a Result-like sealed type for domain errors
sealed interface ChargeResult permits Charged, Declined, GatewayDown {}
record Charged(String txId) implements ChargeResult {}
record Declined(String reason) implements ChargeResult {}
record GatewayDown(Duration retryAfter) implements ChargeResult {}

// exceptions remain for programming errors (NPE, IllegalState)
// — that is the correct split

// Result + the ? operator: the reference form of the approach
fn charge(card: &Card, amount: Cents) -> Result<TxId, ChargeError> {
    let token = tokenize(card)?;          // early return, the type stays honest
    gateway::charge(token, amount)        // ChargeError: an enum of every outcome
}

// thiserror for libraries, anyhow for applications: the settled boundary

// (T, error) is the language idiom; the key is wrapping with context and errors.Is/As
func (s *Svc) Charge(ctx context.Context, c Card, amt Cents) (TxID, error) {
    tok, err := s.tokenize(ctx, c)
    if err != nil {
        return "", fmt.Errorf("tokenize card ****%s: %w", c.Last4, err)
    }
    id, err := s.gw.Charge(ctx, tok, amt)
    if err != nil {
        if errors.Is(err, gateway.ErrDeclined) { return "", ErrDeclined }
        return "", fmt.Errorf("gateway charge: %w", err)
    }
    return id, nil
}

# exceptions are the language idiom; fighting it costs more than accepting it.
# Compromise: domain outcomes as values, infrastructure failures as exceptions
from dataclasses import dataclass

@dataclass(frozen=True)
class Declined: reason: str

@dataclass(frozen=True)
class Charged: tx_id: str

def charge(card: Card, amount: int) -> Charged | Declined:
    ...  # Declined is not an exception: it is an expected business outcome
    # ConnectionError stays an exception: it is a failure, not an outcome

Logic is pure functions with no I/O; a thin shell executes the effects. It kills mock hell: if a test contains when(repo.findById(...)), logic is tangled with I/O.

// core: a pure function, table-tested without a single mock
static List<LedgerEntry> settle(List<Trade> trades, FxRates rates, LocalDate day) { ... }

// shell: all I/O lives here, no logic
void settleDaily() {
    var trades = repo.tradesFor(today);          // I/O
    var rates  = fxClient.ratesFor(today);       // I/O
    var entries = Settlement.settle(trades, rates, today);  // pure
    repo.saveAll(entries);                       // I/O
}

// Rust nudges you here itself: a pure core needs neither async nor Send/Sync
fn settle(trades: &[Trade], rates: &FxRates, day: NaiveDate) -> Vec<LedgerEntry> { ... }

async fn settle_daily(db: &Db, fx: &FxClient) -> Result<()> {
    let (trades, rates) = tokio::try_join!(db.trades(today), fx.rates(today))?;
    db.save(settle(&trades, &rates, today)).await
}

// the core is a package with no I/O imports; a deps linter enforces it
package settlement // imports only time and domain types

func Settle(trades []Trade, rates FxRates, day time.Time) []LedgerEntry { ... }

# the I/O-free core is tested with hypothesis; the shell is 5 lines
def settle(trades: list[Trade], rates: FxRates, day: date) -> list[LedgerEntry]:
    ...  # no await, open(), requests — by construction

async def settle_daily(db: Db, fx: FxClient) -> None:
    trades, rates = await asyncio.gather(db.trades(TODAY), fx.rates(TODAY))
    await db.save(settle(trades, rates, TODAY))

Haskell turns this separation from discipline into law: every side effect is marked with the IO type, and the compiler won’t let you call it from a pure function — there, functional core / imperative shell is not a style but a consequence of the type system. (A precision along the way: monads are not “just for I/O”; Maybe, List, State are monads too, IO is merely the best known.) In Java/Go/Python the boundary is held by convention and a linter — which is what the examples show.

A value that cannot be changed cannot be changed behind your back either: a whole class of “who else holds this reference” questions disappears, and sharing across threads no longer requires locks. The price is copying on change, and it is paid at the boundary: a collection a constructor accepts gets copied, otherwise the caller keeps a lever on your state.

The idea is old in functional programming and was brought to mainstream object-oriented code by Joshua Bloch’s Effective Java (“Minimize mutability”: a class should be immutable unless there is a very good reason to make it mutable). The data-structure side — sharing structure between versions instead of copying everything — goes back to Driscoll, Sarnak, Sleator and Tarjan’s “Making Data Structures Persistent” and to Okasaki’s Purely Functional Data Structures; it is why Clojure or Scala can “copy” a million-element map by allocating a few nodes. Pat Helland’s “Immutability Changes Everything” (CIDR 2015) shows the same idea at the scale of storage systems.

// record + List.copyOf; collections via immutable factories
public record Portfolio(String owner, List<Position> positions) {
    public Portfolio { positions = List.copyOf(positions); }  // defensive copy

    public Portfolio withPosition(Position p) {
        var next = new ArrayList<>(positions); next.add(p);
        return new Portfolio(owner, next);
    }
}

// immutability is the language default; mut is the explicit exception
let portfolio = Portfolio::new(owner, positions); // can't be changed without mut

// shared state: Arc<Portfolio> is cheap and safe precisely because
// its contents are immutable

// the language has no immutability; by convention small value types are copied,
// and slices are copied explicitly in constructors
func NewPortfolio(owner string, ps []Position) Portfolio {
    return Portfolio{owner: owner, positions: slices.Clone(ps)}
}

# frozen dataclass + tuple instead of list
@dataclass(frozen=True, slots=True)
class Portfolio:
    owner: str
    positions: tuple[Position, ...]

    def with_position(self, p: Position) -> "Portfolio":
        return replace(self, positions=(*self.positions, p))

In data infrastructure it is the default already. A Kafka log is append-only; a Parquet file in an Iceberg or Delta table is never rewritten in place — a commit writes new files and atomically swaps a pointer to a new snapshot. That is what makes time travel, concurrent readers without locks and a cheap rollback possible: the old snapshot simply still exists. Mutation moves to one place (the commit of the pointer) instead of being spread over every file.

Where it stops paying. Shallow immutability is a common false sense of safety: List.copyOf copies the list, not the Position objects in it, and a frozen dataclass holding a list is still mutable through that list. In hot loops over large arrays, copy-on-change costs allocation and GC pressure that data-oriented layouts (10) avoid by mutating in place. And Rust shows that the real hazard is shared mutation, not mutation as such: an exclusive &mut is safe because nobody else can see the change. The rule is “immutable when shared”, and the default leans that way because sharing is hard to see.

The metric: how many files you must open to understand a snippet. An abstraction appears on the third repetition, not the first. An interface with a single implementation costs navigation and buys no flexibility.

// The Go community says it outright: "accept interfaces, return structs" —
// the consumer declares the interface, and only once implementations >= 2

// ANTIPATTERN: an interface with a single implementation — two places to read
type UserService interface {
    QueryUser(ctx context.Context, id UserID) (User, error)
}
type userServiceImpl struct{ db *sql.DB }

// NORM: a direct call, one place to read
func (h *Handler) route(ctx context.Context, id UserID) (User, error) {
    return h.db.QueryUser(ctx, id)
}

// ANTIPATTERN: an interface with exactly one implementation and one caller —
// +1 file, +1 jump while reading, zero substitutability in practice
public interface UserService { User find(UserId id); }
public final class UserServiceImpl implements UserService { ... }

// NORM: call the class directly. The interface arrives with the
// second implementation — or at the I/O boundary, where a test needs it (5)
public final class UserService { public User find(UserId id) { ... } }

// ANTIPATTERN: dyn for a single implementation — an indirect call
// at runtime and a jump across files while reading
struct Handler { store: Box<dyn Store> }

// NORM: a concrete type while there is one implementation. When a second
// appears, a generic: monomorphization gives flexibility at no runtime cost
struct Handler { store: PostgresStore }

# ANTIPATTERN: an ABC hierarchy for a single subclass
class Store(ABC):
    @abstractmethod
    def get(self, key: str) -> bytes | None: ...

class PostgresStore(Store): ...     # the only implementation

# NORM: the class directly. When there are two, a Protocol: it needs no
# inheritance and is declared by the consumer, not the implementation (12)
class PostgresStore:
    def get(self, key: str) -> bytes | None: ...

Precondition, postcondition, invariant — and the main thing that follows from them: who is at fault. A violated precondition is the caller's bug: it can occur in a correct program, is always checked, and is answered with an exception. A violated invariant is your own bug and a sign of already-corrupted state: you cannot carry on, hence assert or panic rather than a tidy error return (25).

The term and the discipline are Bertrand Meyer’s: Eiffel builds require, ensure and invariant into the language, and his “Applying ‘Design by Contract’” (IEEE Computer, 1992) and Object-Oriented Software Construction set out the method. The logic underneath is Hoare’s 1969 “An Axiomatic Basis for Computer Programming”: a triple {P} S {Q} says that if P holds before S, Q holds after. Meyer added the part that matters for teams — a contract is a division of responsibility. The caller guarantees the precondition, so the callee need not defend against it twice; the callee guarantees the postcondition, so the caller need not recheck it. His rule for subclasses — a precondition may only be weakened, a postcondition only strengthened — is the precise form of Liskov substitution (1).

// Objects.requireNonNull + explicit invariants in the constructor; assert for internal ones
public record Transfer(Account from, Account to, Cents amount) {
    public Transfer {
        Objects.requireNonNull(from); Objects.requireNonNull(to);
        if (amount.value() <= 0) throw new IllegalArgumentException("amount must be > 0");
        if (from.equals(to)) throw new IllegalArgumentException("self-transfer");
    }
}

// debug_assert! on internal invariants — free in release
fn apply(&mut self, entry: LedgerEntry) {
    debug_assert!(self.is_balanced(), "ledger invariant broken before apply");
    // ...
    debug_assert!(self.is_balanced(), "apply broke double-entry invariant");
}

// TigerBeetle style — cheap invariant checks stay on in production
func (l *Ledger) Apply(e Entry) error {
    if l.debits != l.credits { // double-entry invariant, always on
        panic("ledger corrupted: unbalanced before apply") // corrupted state = crash-only
    }
    ...
}

# assert for invariants (stripped by -O), exceptions for preconditions
def apply(self, entry: LedgerEntry) -> None:
    if entry.amount <= 0:
        raise ValueError("precondition: amount > 0")   # caller's bug
    assert self._is_balanced(), "invariant broken"     # my bug

The ledger example, and its costs. “Debits equal credits” is the invariant of a double-entry ledger; checking it before and after every apply turns silent corruption into a crash at the line that caused it, which is the whole point. TigerBeetle’s style guide keeps dense assertions on in production for that reason. Whether you can afford that is a measurement: an O(1) balance check is free, an O(n) scan of the ledger on every write is not. Know what your platform does by default — Java assert is off unless the JVM runs with -ea, Python strips assert under -O, Rust’s debug_assert! disappears in release — so a check that must always run is written as an ordinary if with a panic.

Where contracts end. Across a process or network boundary the caller is not a cooperating part of your program, so a precondition there is not “the caller’s bug” but input to parse (2, 32) and an error to return (4). A precondition the type system can carry should move into the type (3): a PositiveCents needs no runtime check at each call.

An example test checks the cases you thought of; a property test checks a claim that must hold for every input, and a generator hunts for the counterexample — and, once it finds one, shrinks it to a minimal one. Properties that pay off almost everywhere: roundtrip (parsing what was written yields the original), conservation (the sum does not change), agreement with a naive reference implementation. Types are the same idea one step earlier: what the compiler guarantees (3) no longer needs testing.

The method comes from Koen Claessen and John Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs” (ICFP 2000): the programmer states properties, the library generates random inputs, and a failing input is shrunk to a small counterexample. Hypothesis (Python), jqwik (Java) and proptest (Rust) descend from it. Coverage-guided fuzzing (AFL, libFuzzer, Go’s built-in fuzzing) is the same loop with a different generator: instead of a distribution you describe, it mutates inputs that reach new code paths.

// a property instead of examples
@Property
void settlementConservesMoney(@ForAll("trades") List<Trade> trades) {
    var entries = Settlement.settle(trades, RATES, DAY);
    assertThat(sumDebits(entries)).isEqualTo(sumCredits(entries));
}

// roundtrip: the cheapest property, and the one that catches the most
proptest! {
    #[test]
    fn serde_roundtrip(cfg in any::<Config>()) {
        let json = serde_json::to_string(&cfg).unwrap();
        prop_assert_eq!(serde_json::from_str::<Config>(&json).unwrap(), cfg);
    }
}

// testing/quick is obsolete; the idiom is rapid or built-in fuzzing
func FuzzParseEmail(f *testing.F) {
    f.Fuzz(func(t *testing.T, raw string) {
        e, err := ParseEmail(raw)
        if err == nil && !strings.Contains(e.String(), "@") {
            t.Fatalf("parsed invalid email: %q", raw) // the parser accepted garbage
        }
    })
}

from hypothesis import given, strategies as st

@given(st.lists(trade_strategy()))
def test_settlement_conserves_money(trades):
    entries = settle(trades, RATES, DAY)
    assert sum_debits(entries) == sum_credits(entries)

The strongest form for backend code is a model. A stateful property test runs random sequences of operations — put, delete, compact, restart — against the real component and against a trivial model (a dict), and asserts after every step that they agree. That is how bugs that need five specific operations in a row get found: nobody writes that example by hand. Hypothesis ships this as RuleBasedStateMachine; the same idea at cluster scale is checking histories against a model (98).

Limits. A property test samples; it is not a proof, and a generator that never produces the dangerous shape (empty lists, duplicate keys, Unicode, boundary timestamps) passes forever. Weak properties (“does not throw”) catch little; the work is in finding a property that is strong and still easy to state, and for some domains — rendering, ranking, anything judged by taste — there is none, and examples remain the right tool. Pin the seed and keep found counterexamples as regression cases, or a failure seen once in CI will not come back on demand.

Design driven by data layout: SoA instead of AoS, cache lines, batches instead of per-element processing. It runs against encapsulation — a deliberate trade-off for the hot path.

AoS (array of structs) is an array of records, the way a human writes them; SoA (struct of arrays) is one array per field. The difference shows when a loop reads one field across a million records: the CPU pulls memory in 64-byte lines, so AoS drags the whole record into cache and throws away almost the entire line, while SoA places exactly what is being read side by side. Hence the conflict with encapsulation (12): SoA takes the object apart and strips it of the right to hide its representation — which is why it is applied surgically, on a measured (37) hot path, not across the whole codebase.

// until Valhalla, objects mean pointers and cache misses.
// hand-rolled SoA + primitive arrays, or MemorySegment (FFM API)
final class Particles {           // instead of List<Particle>
    final float[] x, y, vx, vy;   // 4 dense arrays, SIMD-friendly

    void step(float dt) {
        for (int i = 0; i < x.length; i++) { x[i] += vx[i] * dt; y[i] += vy[i] * dt; }
    }
}

// SoA + iterators compile down to SIMD; #[repr(C)] controls layout
struct Particles { x: Vec<f32>, y: Vec<f32>, vx: Vec<f32>, vy: Vec<f32> }

impl Particles {
    fn step(&mut self, dt: f32) {
        for (x, vx) in self.x.iter_mut().zip(&self.vx) { *x += vx * dt; }
        for (y, vy) in self.y.iter_mut().zip(&self.vy) { *y += vy * dt; }
    }
}

// same idea — slices of primitives instead of []*Particle
// (removes pointer chasing and GC pressure)
type Particles struct{ X, Y, VX, VY []float32 }

# DOD = "get out of the interpreter" — numpy/arrow, vectorize instead of looping
import numpy as np

class Particles:
    def __init__(self, n): self.x, self.vx = np.zeros(n), np.zeros(n)
    def step(self, dt): self.x += self.vx * dt   # one C operation instead of n iterations

# polars/duckdb for tabular data — same principle one level up

Every resource has exactly one owner responsible for releasing it, and the release point is tied to scope, not to programmer discipline. Languages differ only in who enforces this: Rust checks ownership in the compiler; the rest provide a construct (try-with-resources, defer, with) and catch what was forgotten with a linter. A garbage collector does not fill this role: it manages memory, not file handles, connections, and locks, and the moment of finalization is undefined.

The idiom has a name from C++: Bjarne Stroustrup’s “resource acquisition is initialization” (RAII) — acquire in a constructor, release in the destructor, and let scope exit, including exit by exception, run the release. Rust turned the idiom into a checked rule: every value has one owner, ownership moves explicitly, borrows are tracked by the compiler, and Drop runs when the owner goes out of scope. Java’s try-with-resources, C#’s using, Go’s defer and Python’s with give the same release point without the compiler proof.

// the reference: ownership in the type; the compiler proves no use-after-free
fn process(conn: Connection) -> Report { ... }
// took ownership: the caller can no longer use conn

// try-with-resources + AutoCloseable; Cleaner for native resources
try (var conn = pool.acquire(); var tx = conn.begin()) {
    ...
} // closed in reverse order, guaranteed

// defer right after acquire is the idiom; a linter catches a forgotten defer
conn, err := pool.Acquire(ctx)
if err != nil { return err }
defer conn.Release()

# context manager; ExitStack for several resources
with pool.acquire() as conn, conn.begin() as tx:
    ...

How it fails in a service. A handler takes a connection from the pool, an exception skips the release, and the pool leaks one connection per error; under an error spike the pool empties and the service stops serving requests that have nothing to do with the original bug. A transaction that is never closed is worse: in PostgreSQL a session left “idle in transaction” keeps its snapshot, holds back vacuum and can keep row locks. Both are scope bugs, and both disappear when the release is tied to the block rather than to the happy path.

Where scope is the wrong shape. Some lifetimes are not lexical: a connection handed to a background task, a lease renewed across requests, a pool that lives as long as the process. Then ownership has to be transferred explicitly and the new owner named, not left to whoever last touched it. Traps of the idioms themselves: Go’s defer inside a loop runs at function exit, not at the end of the iteration, so a loop over files holds every descriptor until it returns; finalizers and Java’s Cleaner are a safety net for a forgotten release, never the release mechanism.

Shared-memory correctness is two separate questions. Visibility: is there a happens-before edge (a lock, a volatile write, an atomic operation, a channel send) from the write to the read, or may the reader see a stale or half-built value? Atomicity: does every invariant-bearing sequence — read-modify-write, check-then-act — run as one step? A “thread-safe” collection answers the first question for each call and neither question for a pair of calls; freedom from data races, which the Go memory model, the JLS and the Rustonomicon each define, is necessary and not sufficient.

final class SessionRegistry {
    private final ConcurrentMap<String, Session> sessions = new ConcurrentHashMap<>();
    private final Function<String, Session> loader;
    private volatile long hitsRacy;              // every write is visible to later reads...
    private final LongAdder hits = new LongAdder();

    SessionRegistry(Function<String, Session> loader) { this.loader = loader; }

    Session getRacy(String id) {
        hitsRacy++;                              // BUG: ...but read-add-write is three steps
        Session s = sessions.get(id);            // each call is thread-safe, the pair is not:
        if (s == null) {                         // two threads both miss, both load,
            s = loader.apply(id);                // and hand out different sessions
            sessions.put(id, s);
        }
        return s;
    }

    Session get(String id) {
        hits.increment();
        // FIX: check-then-act is one atomic map operation; concurrent callers for an
        // absent key wait for a single load. The loader must be short and must not
        // touch this map.
        return sessions.computeIfAbsent(id, loader);
    }
}

type Registry struct {
    mu       sync.Mutex
    sessions map[string]*Session // guarded by mu
    hits     atomic.Int64
    load     func(string) *Session
}

// BUG: no happens-before edge between goroutines touching the map. That is a
// data race (go test -race reports it; the runtime may abort with "concurrent
// map writes"), and even a race-free map would let two goroutines both miss.
func (r *Registry) GetRacy(id string) *Session {
    s, ok := r.sessions[id]
    if !ok {
        s = r.load(id)
        r.sessions[id] = s
    }
    return s
}

// FIX: one critical section spans the check and the act.
func (r *Registry) Get(id string) *Session {
    r.hits.Add(1) // a lone counter needs no lock, only an atomic step
    r.mu.Lock()
    defer r.mu.Unlock()
    s, ok := r.sessions[id]
    if !ok {
        s = r.load(id)
        r.sessions[id] = s
    }
    return s
}

// Compiles: safe Rust rules out the data race, not the lost update.
fn withdraw_racy(balance: &Mutex<u64>, amount: u64) -> bool {
    let current = *balance.lock().unwrap(); // guard is dropped right here
    if current < amount {
        return false;
    }
    *balance.lock().unwrap() = current - amount; // BUG: writes back a stale read
    true
}

// FIX: one guard spans the check and the act.
fn withdraw(balance: &Mutex<u64>, amount: u64) -> bool {
    let mut guard = balance.lock().unwrap();
    if *guard < amount {
        return false;
    }
    *guard -= amount;
    true
}

// Lock-free: compare-and-swap in a loop; the closure re-runs on a lost race,
// so it must be pure. A plain number has no ABA problem; a recycled pointer does.
fn withdraw_cas(balance: &AtomicU64, amount: u64) -> bool {
    balance
        .try_update(Ordering::SeqCst, Ordering::SeqCst, |cur| {
            cur.checked_sub(amount)
        })
        .is_ok()
}

import threading


class Registry:
    def __init__(self, load):
        self._load = load
        self._sessions = {}
        self._hits = 0
        self._lock = threading.Lock()

    def get_racy(self, sid):
        self._hits += 1  # BUG: load, add, store; no build promises they stay together
        if sid not in self._sessions:  # check ...
            self._sessions[sid] = self._load(sid)  # ... act: two threads both load
        return self._sessions[sid]

    def get(self, sid):
        # FIX: one lock spans the counter and the check-then-act. The free-threaded
        # build locks each dict operation internally; that protects the dict,
        # never a sequence of operations on it.
        with self._lock:
            self._hits += 1
            session = self._sessions.get(sid)
            if session is None:
                session = self._sessions[sid] = self._load(sid)
            return session

Visibility. Without a happens-before edge the compiler and the CPU may reorder or cache, and the JLS gives the canonical surprise: a reader that sees a non-null reference is guaranteed the constructor’s value only for final fields; a plain field may still read as zero. Hence safe publication: make the object immutable (6), or hand the reference over through a synchronized channel — a lock, a volatile or atomic reference, a concurrent collection, sync.Once — and never let this escape from a constructor. Double-checked initialization on a plain flag is broken in Go for the same reason: seeing done == true says nothing about seeing the value it guards. In Go a race on a map, slice, string or interface can corrupt memory, not merely lose an update; in Java the damage stays within the language’s memory safety but the observed behaviour can be arbitrarily strange. Run go test -race in CI — it finds races that executions actually hit, not all of them.

Atomicity. The Rust pane compiles: Send/Sync and the borrow checker exclude data races, and the Rustonomicon states plainly that general race conditions remain the programmer’s problem. The Python pane is the same bug on both builds: the GIL never promised that += or check-then-act are atomic, it only made the interleaving rare, and PEP 703’s per-object locks protect one container operation, not a sequence. The decision rule: name the invariant, then give its whole read-and-write one critical section, one atomic primitive (computeIfAbsent, compare-and-swap), or one owner thread that the others message. This is write skew (90) inside a process; for a check-then-act across processes see 66, and for many callers loading the same key see 71.

The primitives carry their own traps. A CAS loop re-runs its update function after every lost race, so the function must be pure; and CAS compares values, not histories — a pointer freed and reused between the read and the swap passes the comparison (ABA), which is why lock-free structures over recycled memory need hazard pointers, epochs or tagged versions. Locks trade that for deadlock: whenever code holds two locks, acquire them in one global order, and never call unknown code (a callback, a loader, a logger that blocks) while holding one — the computeIfAbsent contract forbids exactly that for its mapping function.

A module hides a decision that is likely to change. The criterion is testable: “what breaks if this decision is revisited”. SRP (1) is a blurry retelling of this principle.

A secret is a concrete decision: storage format, encoding, field order, algorithm. An interface that returns “a list of ASCII strings” has already leaked the encoding — moving to UTF-8 breaks every consumer. The same interface described as “grocery item names” would have survived that move unnoticed. The difference is not in the code but in what is declared to the outside.

Why Java needed a separate tier of visibility. Before the module system, public meant “visible to the whole classpath”: the language offered no way to hide a decision between packages of one library, and a helper class made public for a neighbouring package became, by Hyrum’s Law (16), someone else’s contract. exports adds a second tier: public and exported is API; public in a non-exported package does not exist from the outside. Nothing changes inside the module — packages still see each other’s public types, exported or not, and private/protected/public work as before; the module merely filters which part of the public surface is presented to the world. The file’s other directives have nothing to do with hiding: requires defines the dependency graph, opens selectively permits reflection, uses/provides wire up ServiceLoader.

Three words next to module path. Classpath is a list of locations on disk (directories and JARs) where the JVM may look for classes: launch configuration, not a mechanism, and it is flat — hence the whole story above. Module path is the same list, but with module-info read; only there does the second tier of visibility appear. ClassLoader is what finds bytecode by class name and loads it into the JVM, lazily, on first use; a custom loader is a unit of isolation (Tomcat gives one to each application, Flink to each job, child-first, so a job can bring its own library versions), and one class loaded by two loaders is two different types to the JVM: hence ClassCastException “between identical classes”. ServiceLoader means “find all implementations of an interface whose names I don’t know at compile time”: that is how JDBC finds drivers; uses/provides (88) is its contract in the module system, and before modules a text file in META-INF/services did the same job.

// module system (module-info.java) — hiding at the artifact level
module billing.core {
    requires transitive billing.model;  // visible to consumers: part of the API
    exports com.acme.billing.api;       // API only

    // com.acme.billing.internal is neither exported nor opened:
    // normal access is denied, and so is deep reflection (setAccessible, the
    // historical bypass of any modifier). Access can still be granted explicitly —
    // opens, Module.addOpens(), --add-opens: this is a default boundary,
    // not a physical impossibility.
}

// pub(crate) / private by default — hiding is built into the language
pub struct RateLimiter { /* fields are private */ }

pub(crate) mod internal;   // visible inside the crate, invisible to consumers

// internal/ package — the compiler forbids imports from outside the subtree
// myservice/internal/ratelimit — physically unreachable from other modules

# _prefix + __all__ convention; no enforced hiding — and that's honest
__all__ = ["RateLimiter"]   # the public contract is declared explicitly

The only pair of design metrics that can actually be measured (afferent/efferent coupling, instability). In practice: dependency direction is checked in CI, not in the reviewer's head canon.

Both words come from Stevens, Myers and Constantine, “Structured Design” (IBM Systems Journal, 1974), which ranked kinds of coupling and cohesion and argued for low coupling between modules and high cohesion within them. The countable metrics are later: Robert C. Martin’s package metrics (afferent and efferent coupling, instability, abstractness) and the Stable Dependencies Principle — depend in the direction of stability.

What exactly is counted: Ca (afferent) — how many modules depend on you; Ce (efferent) — how many you depend on; instability I = Ce / (Ca + Ce). I = 0 — everyone depends on you and you depend on no one: expensive to change, but nobody breaks you. I = 1 — the reverse. The value is not in the numbers themselves but in the direction: the stable must not depend on the unstable, otherwise a change in the most volatile module propagates up the graph to all its readers. Cohesion is not measured with the same ruler — it is the question “does the module's content change for a single reason” (1), and it remains a judgment call.

// dependency rule as a test; definedBy matches on PACKAGE NAMES —
// ArchUnit reads bytecode, so the naming convention is this test's contract
@ArchTest
static final ArchRule layering = layeredArchitecture().consideringAllDependencies()
    .layer("domain").definedBy("..domain..")
    .layer("infra").definedBy("..infra..")
    .whereLayer("domain").mayNotAccessAnyLayer(); // domain knows nothing about infra

Go — depguard / go list -deps in CI. Rust — crates in a workspace (dependency graph = Cargo.toml, cycles impossible). Python — import-linter with a layers contract.

What the import graph does not see. Two services with no code dependency can be tightly coupled through a shared database table, a message format, a cache key layout or an assumption about ordering: change one column and both break, and no ArchUnit rule notices. Meilir Page-Jones’s connascence names these hidden kinds — of name, type, meaning, timing — and ranks them by how far apart the coupled pieces live. The practical rule: a dependency that crosses a team or a deploy unit must be visible as a contract (18, 74), because the graph will not show it. Cohesion, in turn, trades against DRY (43): pulling shared code into a common module raises coupling to it, and two copies that change for different reasons are more cohesive than one shared helper.

No model is universal. Billing’s “Customer” and marketing’s “Customer” are different types; merging them produces a god table with 60 nullable columns.

Where the problem comes from. In an online store four departments say “order”, and each means something different: the warehouse means line items and shelf locations, accounting means the amount, tax and payment status, shipping means the address and box dimensions, support means the conversation with the customer. As long as this is a single Order class, it has to carry all four departments’ fields at once, and every consumer leaves three quarters of them empty — which is exactly how you end up with a table of sixty nullable columns. Worse than the columns is what gets glued together: the departments have independent reasons to change (1), and a change to the shipment format now passes through a type that tax calculation depends on.

What a boundary is. A bounded context is a region of the system inside which every word means exactly one thing, and on the way out of which meaning has to be translated. Billing’s “Customer” and marketing’s “Customer” are not two versions of one concept but two different concepts that happen to share a word: a river bank and a savings bank do not become one thing because their names match. Matching fields prove nothing — both have id, email, name, but they will change for different reasons and on different days. The criterion is the same as Parnas’s (12): the boundary follows a decision that gets revisited separately — and for the same reason DRY (43) does not apply across it.

// Different crates, different types; translation at the boundary is explicit

// crates/billing/src/lib.rs
pub struct Customer { pub id: CustomerId, pub payment_methods: Vec<PaymentMethod> }

// crates/crm/src/lib.rs
pub struct Customer { pub id: CustomerId, pub segments: Vec<Segment> }

// Anti-corruption layer: no From<crm::Customer> on purpose — mapping only through
// an explicit function carrying business decisions, not a mechanical one

Translation is a business decision, not field shuffling. The layer that performs it is called an anti-corruption layer: it exists so that another context’s concepts do not leak in. The absence of From<crm::Customer> in the example is deliberate — an automatic conversion pretends the types are the same, and the very first question, “does marketing consider a customer blocked for debt to be active?”, shows that they are not. Such a question is answered not by a mapper but by whoever owns the context.

Three mistakes at the door. A bounded context is not a microservice: boundaries of meaning are drawn in code, and four contexts live perfectly well in one process and one codebase; splitting them into services is a separate decision with a separate price (13). An identical set of fields is not proof of a single type but today’s coincidence, and the axis of change has not been established yet (42). A convenient Customer.from(...) is the most expensive of the three: it is not forbidden, but it creates the impression that translation is mechanical, and thereby hides the place where a decision had to be made.

Architecture converges on the organization’s communication structure. Not advice but an observation: if you want a different architecture, restructure the teams first (the inverse Conway maneuver) — otherwise the old org chart will reassert itself in the interfaces.

The source is Melvin Conway, “How Do Committees Invent?” (Datamation, April 1968): organizations which design systems are constrained to produce designs which are copies of their communication structures. His argument is structural, not statistical: a design is split into parts before anyone knows the right split, each part goes to a group, and an interface between two parts can only be as good as the communication between the two groups. Fred Brooks named it “Conway’s law” in The Mythical Man-Month (1975). Later empirical work treats it as a hypothesis to test: Herbsleb and Grinter (ICSE 1999) found coordination breaking down exactly at cross-site interfaces, and MacCormack, Rusnak and Baldwin (Research Policy, 2012) found loosely coupled open-source organizations producing more modular code than tightly coupled commercial ones on matched products. Supportive, but correlational.

A data-platform example. Ingestion, transformation and serving belong to three teams; the result is three systems joined by two hand-offs — a raw bucket and a warehouse schema — and every change to an event needs three tickets. When the company reorganizes around domains (orders, payments) with one team owning an event from producer to served table, the pipeline gets cut the same way within a year, whether or not anyone redraws the architecture diagram. That is the inverse Conway maneuver, a name popularized by Thoughtworks and by Skelton and Pais’s Team Topologies (2019): pick the boundaries you want in the code (14), then give each boundary one owning team.

Limits. It is a tendency with friction, not a law of nature: a small team can build a well-modularized system, and a reorg does not refactor code by itself — it only changes which refactors become cheap. Misuse is reorganizing as a substitute for design: teams cut along the wrong seams produce the wrong seams faster. It trades against Brooks’s observations (76): every new team boundary is also a new communication cost. Not expressible in code; expressed in the org chart.

A type from someone else's library that ends up in your public signature becomes your contract: consumers compile against it, and your upgrade of that library is their breaking change. The leak is silent because no compiler asks “do you really want to expose this?”; per Hyrum's Law (16), the answer turns up when it is already too late to break anything.

The check is mechanical: list every type mentioned in the module's public surface. Anything that is not in your own code is someone else's contract you have silently taken on supporting. Then comes a choice of three, and it must be deliberate: wrap the foreign type in your own (the dependency then stays a secret, 12), declare the transitivity explicitly and accept it as part of the API, or remove the type from the signature. The worst option is the fourth, the default one: choosing nothing.

module billing.api {
    // I need it, consumers can't see it: an implementation detail
    requires com.fasterxml.jackson.databind;

    // a type from billing.model appears in my public signatures,
    // so consumers must be able to see it. transitive is an admission:
    // “billing.model compatibility is now my problem”
    requires transitive billing.model;

    exports com.acme.billing.api;
    exports com.acme.billing.spi to billing.plugins;  // qualified export

    // reflection is granted narrowly and by name, not to all comers
    opens com.acme.billing.dto to com.fasterxml.jackson.databind;

    // the extension point is declared, not found by scanning the classpath
    uses com.acme.billing.spi.TaxProvider;
    provides com.acme.billing.spi.TaxProvider with com.acme.billing.tax.DeTaxProvider;
}

// pub use: same trick, same price; a foreign type becomes your API,
// and a major bump of chrono becomes a major bump of your crate
pub use chrono::NaiveDate;

// a boundary instead of a leak: the newtype keeps chrono an implementation detail
pub struct SettlementDate(NaiveDate);

// a pub field of a pub struct is a contract too; non_exhaustive keeps
// the right to add fields without breaking consumers
#[non_exhaustive]
pub struct Settlement { pub date: SettlementDate }

// Go has neither requires transitive nor re-exports: transitivity leaks
// through the signature, the only place it is visible at all. The consumer
// has to import lib/pq to inspect this error
func Charge(ctx context.Context, id string) (*pq.Error, error)

// the boundary: the foreign error is wrapped in ours; the contract is errors.Is
var ErrDuplicate = errors.New("billing: duplicate charge")

func Charge(ctx context.Context, id string) error {
    if err := insert(ctx, id); isUniqueViolation(err) {
        return fmt.Errorf("charge %s: %w", id, ErrDuplicate) // lib/pq stays inside
    }
    return nil
}

# the same obligation, only declared by convention rather than by the compiler
from ._settlement import Settlement
from ._vendor_client import Client as Client  # redundant alias = “this is API” (PEP 484)

# a second, more explicit layer of the same statement: Client is promised publicly,
# so a vendor upgrade is our breaking change, not theirs
__all__ = ["Settlement", "Client"]

The flip side of the same principle is extension points. uses/provides declare a service in the module contract instead of scanning the classpath at startup: the implementation is plugged in late, but the fact of extensibility is declared early and visible in the artifact. The same choice comes up wherever there are plugins: the set of extension points is part of the public surface, and adding them is cheaper than removing them.

With enough consumers, every observable property of an API is part of the contract: JSON key order, latency, error message text.

// proverb level: don't let anyone observe what you don't promise
// map deliberately randomizes iteration order: the language defends against Hyrum's Law
for k := range m { ... } // order differs on every pass; nothing to rely on

// same here: Set.of()/Map.of() randomize iteration order across JVM runs
// precisely so that nobody starts depending on it

Precision. The Go spec says map iteration order is unspecified and not guaranteed to be the same from one iteration to the next, not that it is “random and always different.” The practical conclusion is the same: you must not depend on it. But the word “randomizes” describes the implementation, not the language contract.

The mechanisms also differ in strength. The Go runtime picks random offsets every time an iterator is initialized: two iterations over the same unmodified map in one process yield different orders. Java computes SALT once at JVM startup (ImmutableCollections; the source comment names the goal outright: “iteration order will vary between JVM runs”): within a run the order is stable, across runs it differs. The contract is the same for both: order is unspecified. What differs is how insistently the implementation keeps you from relying on it, and that itself may change in the next release, which is the main argument for not relying on it.

The only correct protocol for changing a schema or an API under continuous deployment. Never “rename column” — always a sequence in which, at every step, old and new code run side by side.

The full cycle: 1) expand — a nullable column and compatible code; 2) dual write; 3) batched backfill with a checkpoint; 4) reconciliation of old and new values; 5) switching reads; 6) an observation period; 7) NOT NULL and the constraint; 8) contract — removing the old.

-- 1. EXPAND: a nullable column with no default is a fast metadata-only change.
--    NOT NULL and a backfill at this step would take ACCESS EXCLUSIVE on the table.
ALTER TABLE users ADD COLUMN email_normalized TEXT;

-- 2. DUAL-WRITE is the application's job: both fields written in one transaction.

-- 3. BACKFILL in batches with a checkpoint, not a single UPDATE:
--    one big UPDATE = long locks, WAL volume blowup, bloat,
--    replica lag, and no way to interrupt the job.
WITH batch AS (
    SELECT id FROM users
    WHERE email_normalized IS NULL AND id > :checkpoint
    ORDER BY id LIMIT 5000 FOR UPDATE SKIP LOCKED
)
UPDATE users u SET email_normalized = lower(u.email)
FROM batch b WHERE u.id = b.id
RETURNING u.id;   -- the max returned id becomes the new :checkpoint

-- 4. VERIFY: there must be no mismatches; this gates the move to step 5
SELECT count(*) FROM users
WHERE email_normalized IS DISTINCT FROM lower(email);

-- 5-6. Reads switch via a flag (72); dual write stays on for the whole observation period.

-- 7. The constraint is added in two steps to avoid holding a long lock
ALTER TABLE users ADD CONSTRAINT users_email_norm_not_null
    CHECK (email_normalized IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_email_norm_not_null;

-- 8. CONTRACT: only once no live deployment still reads the old field
ALTER TABLE users DROP COLUMN email;

# Same for API fields: the deprecated field lives alongside; return both.
# The marker goes on the field itself, so it reaches JSON Schema and OpenAPI
class UserOut(BaseModel):
    email: str = Field(deprecated="Use email_normalized")
    email_normalized: str

Rollback differs at each step: before step 5 it is enough to stop writing to the new field; after it, switch reads back with the flag. After step 8 there is no rollback — it is a one-way door (20). That is why an observation period is needed between 5 and 8, measured in weeks of deployments, not in hours.

A schema shared by a writer and a reader that deploy separately is a contract with two directions: new readers must read old data (backward compatibility) and old readers must survive new data (forward compatibility). Protobuf and Avro make the safe moves explicit: add, never reuse a field number, reserved for the deleted ones — and a registry turns those rules into a check that fails the build.

All three rules follow from one fact: on the wire a field is identified by its number, not its name — the name lives only in the .proto and in generated code. Hence renaming a field is safe (same number), changing its number is equivalent to replacing the field, and an old reader skips an unfamiliar number as an unknown field — which is why adding is compatible. Hence also the worst case: if a deleted field’s number is given to a new field of a different type, an old reader will parse it with the old schema and silently get someone else’s data — not an error, but plausible garbage. reserved exists precisely so the compiler won’t let you do that; it is the same burned identifier as a principle’s retired number in this reference.

message Order {
  reserved 3, 7;                 // deleted fields: numbers burned forever
  reserved "legacy_status";

  string id = 1;
  int64 amount_cents = 2;
  OrderStatus status = 4;        // adding is the only safe operation
}

// a consumer must survive unknown fields (forward compat)
ObjectMapper mapper = JsonMapper.builder()
    .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES) // deliberately
    .build();

#[derive(Deserialize)]
#[serde(deny_unknown_fields)]  // at a TRUST BOUNDARY: strictness;
struct AdminCommand { ... }    // on a service-to-service contract: the opposite,
                               // tolerance of new fields

Where the rules come from. They are not folklore but the formats’ own specifications: the Protocol Buffers language guide (“Updating a message type”) and the Avro specification’s “Schema Resolution” section, which defines exactly how a reader’s schema is matched against a writer’s. Avro has no field numbers — fields are matched by name — so its version of the rule is different: a field added for a new reader needs a default, or old records cannot be read. Confluent Schema Registry encodes the directions as compatibility modes (BACKWARD, FORWARD, FULL, and _TRANSITIVE variants that check against every earlier version, not just the last); the mode follows from who upgrades first — consumers first is backward, producers first is forward.

What a compatibility check cannot see. It checks shape, not meaning. Changing amount from cents to euros, or a timestamp from UTC to local time, passes every registry check and corrupts every reader; such a change is a new field with a new name, migrated with expand — migrate — contract (17). Some changes are wire-compatible and still lossy: widening int32 to int64 parses on both sides, but an old reader truncates large values. And the tolerance for unknown fields shown above belongs on service-to-service contracts; at a trust boundary, strictness wins (19, 32).

“Be liberal in what you accept” gave us XSS, request smuggling, and protocols that cannot evolve. The modern position: strict validation at the boundary, rejection instead of guessing.

The principle is Jon Postel’s, from the early Internet specifications; RFC 793 (TCP, 1981) phrases it as “be conservative in what you do, be liberal in what you accept from others”. In a young network of independent implementations it helped them interoperate despite each other’s bugs. The costs showed up over decades. If receivers accept malformed input, senders never learn it is malformed, the bugs become the de facto protocol (16), and every new implementation has to reproduce them. Eric Allman, “The Robustness Principle Reconsidered” (ACM Queue, 2011), made the security case: liberal parsing is where attackers live.

Request smuggling is the textbook case. A front-end proxy and a back-end server each “tolerate” a request with both Content-Length and Transfer-Encoding, but resolve the ambiguity differently; the attacker’s bytes that the proxy saw as one request body, the back end reads as the start of a second request. Two lenient parsers that disagree are a vulnerability. The current HTTP/1.1 specification (RFC 9112) says such a message ought to be handled as an error.

An honest note on the title. RFC 9413 is titled “Maintaining Robust Protocols” and formally repeals nothing: it shows that liberal acceptance itself becomes a source of ossification and vulnerabilities, and recommends actively managing tolerance rather than banning it. “Repealed” is this page’s editorial position, not the text of the RFC.

# strict mode at the external boundary
class PaymentIn(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")  # "123" will not become 123

    amount_cents: int
    currency: Literal["EUR", "USD"]

dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields()
// unknown field from an external client = 400, not a silent drop

Where tolerance still belongs. Strict is not the same as brittle. Ignoring unknown optional fields in a versioned schema is how producers evolve without breaking consumers (18) — that is designed extensibility, declared in the contract, not guessing. The line: tolerate what the specification says may vary; reject what it says is wrong, and parse into a typed value once, at the edge (2).

Decision speed is inversely proportional to the cost of rollback. Picking a logging library is a two-way door, settled in an hour. Picking a storage format, partitioning or ID scheme is a one-way door and requires an ADR.

The door metaphor is from Jeff Bezos’s 2015 letter to Amazon shareholders: Type 1 decisions are consequential and irreversible or nearly so, and deserve to be made slowly and carefully; Type 2 decisions are changeable, and should be made quickly by individuals or small teams. His warning runs both ways — an organization that grows tends to apply the heavyweight Type 1 process to everything, and becomes slow. The written record for the Type 1 ones is the Architecture Decision Record, proposed by Michael Nygard in “Documenting Architecture Decisions” (2011): a short file with context, decision, status and consequences, kept in the repository next to the code it constrains.

# ADR-014: Partition key for the orders topic

Status: accepted   Date: 2026-03-02

Context: repartitioning a Kafka topic in prod = consumer downtime
         + rebuilding Flink state stores

Decision: partition by customer_id, not by order_id

Consequences: hot keys for large B2B customers;
              mitigated with salted keys at >5% skew

Why the partition key is a one-way door: the key decides ordering guarantees and which consumer instance holds which state (92). Changing it means a new topic, dual writes or a replay (80), and rebuilding every stateful consumer — weeks, not a config change. The same holds for a primary-key scheme exposed in URLs, a public API field’s meaning (16), and anything written to long-lived storage in a format other systems read.

Turn one-way doors into two-way ones where you can. Expand–migrate–contract (17), a facade in front of storage, feature flags and versioned formats all lower the cost of rollback, and with it the process the decision needs. The misuse is the reverse: writing ADRs for library choices, which buries the few irreversible decisions among many reversible ones.

Shipping code and enabling behavior are two different events, separated by a flag. A canary / percentage rollout turns a release from a binary event into a controlled experiment with a rollback metric. A kill switch is a separate kind of flag: not an experiment but a breaker built in advance for degradation (see 55); it lives forever and is tested regularly.

The term “progressive delivery” was coined by James Governor of RedMonk in 2018 for the practices that sit between continuous delivery and full release: canaries, percentage rollouts, flags, automated analysis of the rollout’s metrics. The vocabulary of flags comes from Pete Hodgson’s “Feature Toggles (aka Feature Flags)” on martinfowler.com (2017), which separates release toggles, experiment toggles, ops toggles and permission toggles by how long they live and how dynamic they are — the kill switch above is an ops toggle, and it is the one kind meant to stay.

// a typed flag, with owner and lifetime right in the declaration
@Flag(owner = "team-payments", removeBy = "2026-10-01", kind = RELEASE)
static final BoolFlag NEW_SETTLEMENT = flags.bool("settlement.v2", false);

if (NEW_SETTLEMENT.on(ctx.customerId())) { ... } // percentage by a stable hash of id,
                                                 // not random(), or the user
                                                 // flickers between versions

An honest assessment. Flag debt is real: every flag means 2ⁿ configurations, of which two get tested. A removal date is part of the definition of done; a flag older than two quarters with no decision is a process incident. The consequence for schemas: a canary means two versions of the code are always live at the same time, so expand — migrate — contract (17) goes from “good practice” to the only option.

Why flag hygiene is not pedantry. The SEC’s 2013 order against Knight Capital describes how, in August 2012, new code reused a flag that had once activated a retired function; one of eight servers had not received the new code, the flag woke the old function there, and the firm lost about $460 million in 45 minutes. A reused flag is the same hazard as a reused field number (18): a name whose old meaning is still alive somewhere.

Where it does not fit. A canary needs enough traffic to tell a regression from noise: a service with a hundred requests an hour gets nothing statistically useful from 1 %, and a staged rollout by time is more honest. Database migrations cannot hide behind a flag at all; they are made safe by being backward compatible (17). And mobile or on-premise clients cannot be rolled back once installed, so for them the flag, not the deploy, is the only rollback there is.

A big-bang rewrite violates Gall’s Law (41) on an industrial scale. Strangler fig: a facade in front of the legacy system, routes moved to the new system one at a time, the legacy dies by displacement. Every step is reversible (route back), progress is measurable (% of traffic), and the system keeps running the whole time.

The name is Martin Fowler’s, from his 2004 bliki entry “StranglerApplication” (later retitled “StranglerFigApplication”), after the fig vines he saw in Australian rainforest: they grow around a host tree until the tree dies and the fig stands on its own. The argument against the alternative is about risk and value: a rewrite delivers nothing until it reaches parity with a system whose behavior is only partly documented, while the old system keeps changing underneath it (76). Strangling delivers each migrated route as soon as it moves, and the parts nobody ever migrates turn out to be the parts nobody needed.

# routing IS the migration mechanism
routes:
  - match: { path: /invoices/* }   # migrated
    backend: billing-v2
  - match: { path: /* }            # everything else: legacy, for now
    backend: legacy-monolith

One route, done properly: put the facade in front first with no behavior change; then shadow /invoices/* — send each request to both, serve the legacy answer, and diff the two responses until the mismatch rate is near zero and every remaining difference is explained; then serve from the new system for a slice of tenants, then all; and only then delete the legacy code for that route. Each step has a metric and a rollback that is a routing change.

The most expensive part is not the routing but splitting the data: while both systems write to one database, the legacy isn’t dead, just hidden. A shared DB is a way station with a departure date; otherwise you’ve built a distributed monolith. Data moves by expand–migrate–contract (17) or change data capture (104), with one system owning writes at any time. Strangling also needs a seam: a batch job with no interface to intercept has to be cut at its inputs and outputs instead.

An E2E environment with every service is the most expensive and least stable way to check compatibility. Consumer-driven contracts: the consumer publishes its expectations (which fields it reads, which statuses it handles), and the provider verifies them in its own CI. A breaking change is caught before deploy, on the provider side, with the name of the affected consumer. This operationalizes Hyrum’s Law (16): the contract is what consumers actually use, recorded in machine-readable form.

The pattern was described by Ian Robinson in “Consumer-Driven Contracts: A Service Evolution Pattern” (martinfowler.com, 2006): a provider’s obligations are the union of what its consumers actually rely on, and making those expectations explicit lets the provider evolve everything else freely. Pact, started at realestate.com.au, turned it into tooling: consumer tests run against a mock that records the interactions as a pact file, a broker stores the pacts per version, and the provider replays them against its real implementation. The broker’s can-i-deploy check answers the release question directly: is this provider version verified against every consumer version in production?

# consumer expectation
pact.given("invoice 42 exists") \
    .upon_receiving("get invoice") \
    .with_request("GET", "/invoices/42") \
    .will_respond_with(200, body=Like({"number": "42", "total_cents": 12300}))

# the provider runs ALL published consumer contracts in its own CI;
# removing a field someone reads = a red build for the provider

Contracts check structure, not behavior — they complement the provider’s tests rather than replace them. A schema registry (protobuf + compatibility checks, 18) does the same for events; contract testing covers boundaries that have no registry (REST/JSON).

Where contracts fail. They protect only consumers who publish one; a consumer that never wrote a pact is invisible, which is Hyrum’s Law again. A public API with unknown clients cannot be contract-tested at all — there, versioning and an OpenAPI diff in CI do the job. Over-specified contracts are the common misuse: a consumer that asserts exact values instead of types and shapes (hence Like above) makes every provider change red, and teams learn to ignore the build.

IaC is not “provisioning scripts” but a declaration of desired state with detectable drift: a manual change in the console is a divergence between code and reality that plan must reveal and the process must forbid creating. terraform plan is a review artifact on a par with a code diff: what gets reviewed is not the HCL but the change plan (“destroy and recreate” on a database is what the reviewer is looking for). The state file is a database, with all that implies: locking, backups; renaming a resource without moved {} = destroy/create — a one-way door (20) disguised as a refactoring.

# non-destructive refactor — a moved block, not “terraform got it from the name”
moved {
  from = aws_msk_cluster.kafka
  to   = module.streaming.aws_msk_cluster.kafka
}

# the invariant lives in code, not in the reviewer's head
lifecycle {
  prevent_destroy = true   # on stateful resources; a plan destroying them fails CI
}

What prevent_destroy does not do. It protects a resource only while that resource is described in the configuration: delete the resource block itself and the protection leaves with it, and the resource gets destroyed. So the protection cannot live in HCL alone. You need: policy-as-code that fails a plan with a destroy on stateful resources; CI that applies the saved plan rather than one recomputed at apply time; a check for deleted resource blocks in the diff; tested backups and restore drills; and an explicit break-glass procedure. Separately: drift is visible only within what the provider is able to read.

“Cattle, not pets” is a rule about how things change, not about the kind of resource: you can replace wholesale whatever is recoverable from code and replicas, while anything holding the only copy of some state requires explicit protocols for migration, restore and version compatibility. Replicated state with a quorum, or object storage, is managed declaratively just like compute; a unique machine with a local disk is not, whatever its tag says. Confusing the two is the standard way to lose state. Drift detection is not a quarterly audit but continuous (plan on a cron with an alert on a non-empty diff): drift found a month later has already grown dependencies — Hyrum (16) for infrastructure.

A zero-downtime migration (ID scheme, storage format, gitflow → trunk, platform) is not a “switch-over” but a pipeline: dual-run (both systems running) → shadow/compare (the new one gets a copy of traffic, discrepancies are measured) → gradual cutover with reversible routing → a coexistence period → contract (tearing down the old one). The progress metric is public (% of traffic / services / records on the new path), and — the most ignored part — the migration has an owner until the end: a migration that is 90% done is worse than one never started, because now there are two systems forever.

// ANTIPATTERN: a goroutine per request — unbounded concurrency,
// context cancelled right after the client reply, no timeout, no recover
go func() {
    nu, nerr := m.next.Get(ctx, id)          // ctx is already cancelled
    m.compare.Record(id, old, err, nu, nerr) // a panic here kills the process
}()

// CORRECT: the new implementation gets traffic but no vote —
// via a bounded queue, with its own context and timeout
func (m *Migrator) Get(ctx context.Context, id UserID) (User, error) {
    old, err := m.legacy.Get(ctx, id)
    m.shadow.Offer(job{id, old, err}) // non-blocking: if the queue is full,
    return old, err                   // the sample is dropped, client doesn't wait
}   // old one is still the source of truth; cutover when diff-rate < threshold
    // for N days straight, and that number is written down BEFORE the start

func (m *Migrator) shadowWorker() {
    for j := range m.shadow.Jobs() {
        func() {
            defer func() { _ = recover() }() // a worker panic doesn't crash the process
            ctx, cancel := context.WithTimeout(m.shadowBase, 200*time.Millisecond)
            defer cancel()
            nu, nerr := m.next.Get(ctx, j.id)
            m.compare.Record(j.id, redact(j.old), j.err, redact(nu), nerr)
        }()
    }
}

Honest assessment. Organizational migrations (gitflow → trunk, monorepo, platform) follow the same mechanics as technical ones: dual-run = both models are allowed, cutover = new services only on the new one, contract = the old one is switched off on a date announced in advance. “Voluntary and open-ended” is not a migration but a permanent doubling of maintenance. The cancellation criterion is also written before the start: a migration without a “when do we roll back” condition is a sunk-cost trap at production scale.

What the code does not show: shadowBase is not a context derived from the request but its own, carrying only the permitted correlation attributes and not a single user attribute; the shadow path needs a circuit breaker, otherwise degradation of the new system starts eating the old one’s resources; and the queue must have a dropped-samples metric — silently lost shadow traffic turns diff-rate into a number that measures itself.

Correctness can only be ensured at the endpoints; intermediate layers provide optimization, not a guarantee. Corollary: a broker's exactly-once ends at the broker's boundary — beyond it, deduplication must live in the consumer.

The precise statement. Kafka transactions provide true EOS in a read-process-write loop where input, output and offsets all live inside Kafka: the offset commit and the result write are atomic. The guarantee disappears exactly where an external sink appears (a database, a payment gateway, an email): it shares no transaction with the broker. So “exactly-once” is not marketing but a scope of applicability, outside which only E2E deduplication works.

-- The dedup marker and the business-state change go in ONE transaction.
-- Otherwise two consumers pass the check at once, or the process crashes
-- between the effect and the marker, and the event is applied twice.
BEGIN;

-- The effect is conditional on the claim: a duplicate inserts no row,
-- so `claimed` is empty and the UPDATE touches 0 rows.
WITH claimed AS (
    INSERT INTO consumed_events (event_id) VALUES (:event_id)
    ON CONFLICT DO NOTHING
    RETURNING event_id
)
UPDATE accounts SET balance = balance + :amount
WHERE id = :account_id AND EXISTS (SELECT 1 FROM claimed);

COMMIT;

void handle(ConsumerRecord<String, OrderEvent> rec) {
    // Dedup and effect inside one storage transaction, not as two
    // calls around process(): the process can die in between.
    tx.run(session -> {
        if (!session.claimEvent(rec.value().eventId())) return; // INSERT .. ON CONFLICT
        applyBusinessChange(session, rec.value());
    });
    // offset is committed AFTER the transaction: a retry is safe, a loss is not
    consumer.commitSync();
}

Where this stops working. The scheme above requires the effect and the marker to live in the same transactional resource. If the side effect is external — a payment gateway, an email, a call to someone else's API — there is no shared transaction, and three options remain: an idempotency key on the external service's side, an outbox (68) followed by reconciliation, or an explicit reconciliation procedure. Deduplication in a separate store with a TTL is an optimization (“don't do the work twice”), not a correctness guarantee.

The idempotency key belongs in the API contract, not in client-side retry logic. Over a network a timeout does not tell you whether the operation happened, so every operation that can be retried must be safe to apply twice.

An operation is idempotent if applying it twice has the same effect as applying it once. HTTP defines this per method (RFC 9110, §9.2.2): PUT and DELETE are idempotent, POST is not. The problem it solves is the indeterminate outcome: the client sent “charge €40”, the connection dropped, and nothing on the client side can tell a lost request from a lost response. Retrying a non-idempotent call risks a double charge; not retrying risks a missing one. Message brokers have the same problem under another name: at-least-once delivery means every consumer sees duplicates, and “exactly-once” in practice means at-least-once delivery plus idempotent processing.

// Idempotency key comes from the client; result is cached atomically with the op
func (s *Svc) CreatePayment(ctx context.Context, key IdempotencyKey, req PaymentReq) (Payment, error) {
    return s.store.WithIdempotency(ctx, key, func(tx Tx) (Payment, error) {
        return s.doCreate(tx, req) // a retry with the same key returns the stored result
    })
}

// The type makes the requirement unbreakable: no key, no request
pub struct CreatePayment { pub idempotency_key: Uuid, pub amount: Cents }

-- same transaction as the payment itself; a unique key, not a read-then-write check
INSERT INTO idempotency (key, request_hash, response)
VALUES ($1, $2, $3)
ON CONFLICT (key) DO NOTHING;

What makes it correct. The key is generated by the client once per logical operation and reused on every retry; Stripe’s Idempotency-Key header is the well-known public example, and an IETF draft (“The Idempotency-Key HTTP Header Field”) standardizes it — still a draft, not an RFC. The server stores the key and the response in the same transaction as the effect, enforced by a unique constraint; a separate “check, then write” races. A reused key with a different request body is an error, not a replay — hence the request hash. Keys expire after a window longer than any client’s retry horizon.

Limits. Idempotency covers the service’s own state. A side effect outside the transaction — an email, a call to a payment provider — needs its own key passed downstream, or an outbox (68). Idempotent is also not commutative: two different updates arriving out of order still need versioning or a fencing token (66).

An unbounded queue turns a failure into a latency collapse (Little's law: L = λW; as the queue grows, response time grows at the same λ).

A buffer and backpressure are not alternatives on a single scale. Overload protection is assembled from four different things: a bounded buffer (absorbs a burst, but not a trend), a concurrency limit and admission control (decide how much work to accept at all), backpressure (pushes the pressure back to where the work came from), and load shedding (fast rejection where there is nowhere to push the pressure). The choice between the last two comes down to one question: can the producer slow down? An internal service on a gRPC stream can; a random user’s browser cannot, and for it the honest answer is 429 with Retry-After, not a thirty-second place in a queue. An end-to-end deadline and a retry budget (24) are part of the same design: without them, shed load comes back as retries and finishes off what you were protecting.

// bounded channel + rejection instead of a growing queue
select {
case jobs <- j:
case <-time.After(50 * time.Millisecond):
    return ErrOverloaded          // load shedding: a fast failure beats a slow success
}

// Reactive Streams: backpressure in the contract; or a bounded executor
var pool = new ThreadPoolExecutor(8, 8, 0, SECONDS,
    new ArrayBlockingQueue<>(1_000),               // bounded!
    new ThreadPoolExecutor.CallerRunsPolicy());    // pressure back onto the producer

// bounded mpsc; send().await is itself the backpressure
let (tx, rx) = tokio::sync::mpsc::channel::<Job>(1_000);
if tx.try_send(job).is_err() { return Err(Error::Overloaded) }

# bounded queue; put() suspends the producer — that is a feature
queue: asyncio.Queue[Job] = asyncio.Queue(maxsize=1_000)
await queue.put(job)   # blocks when full — pressure flows back up the pipeline

Retry without jitter is a distributed metronome: a synchronized crowd finishes off a recovering service.

async fn retry<T, E>(mut f: impl FnMut() -> BoxFuture<'static, Result<T, E>>) -> Result<T, E> {
    let base = Duration::from_millis(100);
    for attempt in 0..5 {
        match f().await {
            Ok(v) => return Ok(v),
            Err(_) if attempt < 4 => {
                let cap = base * 2u32.pow(attempt);
                sleep(Duration::from_millis(rand::random_range(0..cap.as_millis() as u64))).await;
            }
            Err(e) => return Err(e),
        }
    }
    unreachable!()
}

// the context deadline is end-to-end; the caller sets the timeout, not the library
ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
defer cancel()

resp, err := s.gw.Charge(ctx, req) // the whole stack below honors the deadline

// breaker + bulkhead by composing decorators
var decorated = Decorators.ofSupplier(() -> gateway.charge(req))
    .withCircuitBreaker(CircuitBreaker.ofDefaults("gw"))
    .withBulkhead(Bulkhead.of("gw", BulkheadConfig.custom().maxConcurrentCalls(20).build()))
    .withRetry(Retry.of("gw", RetryConfig.custom().intervalFunction(
        IntervalFunction.ofExponentialRandomBackoff()).build()))
    .decorate();

# exponential backoff + jitter, declared declaratively
from tenacity import retry, stop_after_attempt, wait_random_exponential

@retry(stop=stop_after_attempt(5), wait=wait_random_exponential(multiplier=0.1, max=3))
async def charge(req: PaymentReq) -> Payment: ...

What the example above is missing. It retries every error. A production policy distinguishes retryable errors (timeout, 5xx, overload) from non-retryable ones (validation, authentication, 4xx) — retrying the latter is pointless and harmful; honors Retry-After instead of its own backoff; is bounded by an end-to-end deadline, not by a sum of individual timeouts; is cancelled together with the context; has a per-client retry budget so that retries don’t make up most of the traffic; and never retries a non-idempotent operation without an idempotency key (22). Attempt counts and budget exhaustion are metrics; otherwise degradation is invisible.

The only way to stop is to crash; recovery is tested by every deploy. Separate shutdown code goes unexecuted for years and therefore doesn’t work.

The source is George Candea and Armando Fox, “Crash-Only Software” (HotOS IX, 2003). Their observation: a system has two ways to stop — clean shutdown and crash — and must survive the second anyway, because power cuts and kernel panics do not ask. If it must recover from a crash, a separate clean-shutdown path is a second, rarely run, code path to the same place. Make crash the only stop and start the only recovery, and the recovery code runs on every restart, so its bugs are found in daylight. Their requirements for that: all important state lives in dedicated crash-safe stores (a database with a write-ahead log, not process memory), components are restartable on their own, requests carry timeouts and are retried — which needs idempotent operations (22). Erlang’s “let it crash” with supervisor trees is the same idea at the process level.

// The orchestrator (k8s) kills pods constantly — so startup must be recovery
func main() {
    state := mustRecoverFromLog(walPath)   // startup == recovery, always

    // graceful shutdown is minimal: just release leadership / drain the LB;
    // correctness does NOT depend on it
}

A stream consumer example: it commits its offset only after the effect is durable, so a SIGKILL at any instruction means some messages are processed again on restart, never lost — and processing again is safe because the sink is idempotent. A test that kills the process at random points under load (77) checks the property directly.

A clarification. The thesis is that correctness must not depend on graceful shutdown. A clean shutdown in itself remains a useful optimization: draining connections and the load balancer, handing off leadership, reducing rework after a restart. Dropping it is not a consequence of crash-only but a separate decision. The cost is recovery time: a large log replayed on every start makes deploys slow, so checkpoints bound it.

Measure the tail, not the mean: when one user request fans out to many servers, it waits for the slowest of them, so a rare slow response on each server becomes a common slow response for users. Hedged requests attack the tail directly: send a duplicate to another replica once the first has been outstanding longer than the p95, and take whichever answers first.

The source is Jeffrey Dean and Luiz André Barroso, “The Tail at Scale” (Communications of the ACM, February 2013). Their point is that at Google’s scale latency variability cannot be engineered away component by component — background jobs, garbage collection, queueing and shared resources always produce some slow responses — so systems have to be tail-tolerant, the way fault-tolerant systems accept that parts fail. Besides hedging they describe tied requests (both replicas are told about each other and the second cancels once the first starts work), micro-partitioning for fast rebalancing, and putting slow machines on probation.

// hedged request on virtual threads
try (var scope = StructuredTaskScope.open(Joiner.<Reply>anySuccessfulOrThrow())) {
    scope.fork(() -> replica1.get(key));
    scope.fork(() -> { Thread.sleep(p95Latency); return replica2.get(key); }); // hedge
    return scope.join();  // first success wins, the other is cancelled
}

// On JDK 25 (JEP 505) the same joiner was named anySuccessfulResultOrThrow().
// On JDK 21 (JEP 453) the shape differs; the factories did not exist yet:
// try (var scope = new StructuredTaskScope.ShutdownOnSuccess<Reply>()) {
//     scope.fork(() -> replica1.get(key));
//     scope.fork(() -> { Thread.sleep(p95Latency); return replica2.get(key); });
//     scope.join();
//     return scope.result();
// }

The arithmetic and the cost. With a fan-out of 100 independent calls, the probability of hitting at least one component's tail is ≈ 1 − 0.99¹⁰⁰ ≈ 63 %: the component's p99 becomes the user's ordinary experience. Hedging applies only to idempotent, cancellable operations (22) and buys latency with extra traffic, so it needs a budget, and the share of hedged requests belongs in the metrics.

What was measured and what was not. The 63 % figure is arithmetic under an independence assumption; real tails are partly correlated (one GC pause, one noisy neighbour hits many requests at once), so treat it as the shape of the effect, not a prediction. The measured part is the paper’s: in a Google benchmark reading 1,000 keys spread over 100 BigTable servers, hedging after a 10 ms delay cut the 99.9th-percentile latency from 1,800 ms to 74 ms while sending about 2 % more requests. Hedging does not help when the slowness is shared — an overloaded database behind every replica only gets twice the load — and it is the wrong tool for non-idempotent writes.

The unit of failure is designed up front: customers are sharded into cells — complete, independent copies of the stack — and an incident hits one cell, not everyone. Shuffle sharding is a variant that reduces the overlap between customers combinatorially.

A cell has its own compute, its own database and its own queues; the only shared component is a thin router that maps a partition key (tenant, account) to a cell. A bad deploy, a poison message (81), a runaway tenant or a corrupted index stays inside the cell it reached, and deploys roll out cell by cell, so the first cell is the canary for the rest. The idea is the bulkhead (24) applied to the whole stack. AWS describes it in the Well-Architected guidance “Reducing the Scope of Impact with Cell-Based Architecture” and uses it for its own control planes.

Shuffle sharding is described by Colm MacCárthaigh in the Amazon Builders’ Library article “Workload isolation using shuffle-sharding”. Instead of fixed groups, each customer gets a random combination of workers. With 8 workers and 2 per customer there are 28 combinations; a customer whose requests crash their two workers takes down only the customers who share both — on average a small fraction, and clients that retry against the other workers of their shard see little effect. With plain sharding into 4 pairs, the same customer takes out a quarter of everyone.

The cost and the limits. Cells multiply fixed cost (N databases to patch, N dashboards), make cross-cell queries and cross-tenant features hard, and need a migration path for tenants that outgrow a cell. The router and the cell-assignment table are the new single point of failure, so they are kept minimal, cached at the edge and changed rarely. Global dependencies — DNS, the identity provider, a shared config push — bypass cells entirely; an incident there has the old blast radius. Below a few dozen tenants or one region the overhead usually exceeds the benefit. Expressed by deployment topology, not by code.

“We will survive losing a replica” is a hypothesis; until it has been tested by injecting the failure, it is an opinion. The method: steady-state metric → hypothesis (“killing a pod does not move p99”) → minimal blast radius (27) → injection → comparison. The detail people skip: the end goal is production, because staging reproduces neither the traffic, nor the data, nor the configuration.

The practice comes from Netflix: Chaos Monkey, which terminated production instances during business hours, appeared around its move to AWS, and the discipline was written up in Basiri et al., “Chaos Engineering” (IEEE Software, 2016) and in the Principles of Chaos Engineering. The principles are the method above: define steady state as a measurable output, hypothesize that it continues in both the control and the experimental group, vary real-world events, run in production, automate, minimize blast radius.

# an experiment is a versioned artifact, like a test
experiment: kafka-broker-loss
steady_state: { metric: consumer_lag_p99, threshold: "< 30s" }
method: [{ action: kill-pod, selector: "app=kafka, broker=2" }]
rollback: automatic          # abort on threshold breach
blast_radius: cell-eu-1      # one cell, not the fleet

An honest assessment. Chaos without observability is vandalism: if you cannot answer “what broke?”, you have no business breaking anything. The order is strict: SLOs and tracing first, injections second. Game days on staging are a legitimate first step, but not the final one.

What a good experiment finds. Not that the broker died — you knew that — but the assumption around it: a producer configured with acks=1 that loses acknowledged writes on leader failover, a client whose retry storm outlasts the outage (24), an alert that never fired. Each finding becomes a fix plus a permanent automated experiment, like a regression test.

Limits. Fault injection tests resilience, not correctness: a consistency bug under partition needs history checking against a model (98), not a dashboard. Where a failed experiment harms people or breaks a regulatory duty, production injection is replaced by simulation and restore drills (97). And for a small system with one region and an untested backup, the restore test pays off long before the monkey does.

A backup is an artifact; recovery is a capability. RPO (how much data may be lost) and RTO (how long the service may be down) are business numbers set per workload, and neither is a property of a backup job that exits 0. The only evidence for them is a restore that was actually performed, timed end to end and checked for content. Until then “we have backups” is a hypothesis about a procedure nobody has run.

Where the numbers actually come from. Achieved RPO is not the backup schedule but the lag of the last durable, restorable point: for PITR that is the base backup plus an unbroken WAL sequence up to the failure, and a single missing segment ends replay there. RTO is the sum of everything on the path — finding the right backup, fetching it, restoring, replaying, validating, and cutting traffic over — and replay time grows with the distance to the last base backup. Replication does not replace any of this: it carries a DROP TABLE or a corrupting bug to every replica within seconds; the defence against logical damage is a point in time before it. Restoring one store is also not restoring a system: a database and an object store recovered to different moments leave dangling references, the same boundary problem as in 68.

#!/usr/bin/env bash
set -euo pipefail
t0=$(date +%s)                         # the simulated disaster happens now
backup=/backups/latest                 # pg_basebackup -Ft -z --wal-method=stream
pgdata=/drill/pgdata

# 1. Integrity against the manifest (tar format cannot have its WAL parsed, hence -n).
#    Necessary, not sufficient: the PostgreSQL docs still require a real restore.
pg_verifybackup -n "$backup"

# 2. Restore into a scratch instance and replay the archive to its end.
rm -rf "$pgdata" && install -d -m 700 "$pgdata"
tar -xzf "$backup/base.tar.gz" -C "$pgdata"
tar -xzf "$backup/pg_wal.tar.gz" -C "$pgdata/pg_wal"
cat >> "$pgdata/postgresql.auto.conf" <<'CONF'
restore_command = 'cp /archive/%f "%p"'
archive_mode = off   # a promoted drill copy must never write into the production archive
CONF
touch "$pgdata/recovery.signal"      # no recovery target: replay everything, then promote
pg_ctl -D "$pgdata" -o '-p 5499' -l /drill/restore.log -W start
until [ "$(psql -p 5499 -d postgres -Atc 'select pg_is_in_recovery()' 2>/dev/null)" = f ]; do
  sleep 5
done

# 3. Measure the promise instead of asserting it. The RPO figure overestimates
#    the loss when the primary was idle before t0; RTO is wall-clock to a promoted server.
psql -p 5499 -d app -v ON_ERROR_STOP=1 -At <<SQL
select 'rpo_s', extract(epoch from to_timestamp($t0) - pg_last_xact_replay_timestamp());
select 'rto_s', extract(epoch from now()) - $t0;
select 'orders_last_hour', count(*) from orders
 where created_at > to_timestamp($t0) - interval '1 hour';
SQL

Failure modes a drill finds and a dashboard does not. An archive command that returns 0 without having stored the file: PostgreSQL then recycles the segment and the hole surfaces only at restore time. Encryption keys, credentials or the runbook itself living inside the system being recovered. And recovery that depends on the control plane: provisioning instances, changing DNS records, creating resources — APIs that can be impaired while the data plane keeps serving. AWS guidance is explicit here: recover with data-plane actions and pre-provision what recovery needs (101, 105).

Decision rule. A recovery claim is the measured result of the last drill, with its date, for example “on <date>: restored to <N> s before t0, promoted in <M> min, row counts matched” — not a retention setting. Drills run on a schedule, in automation, against the real backup location, and a failed drill pages like an outage, because it is one that has not happened yet. If the measured RTO exceeds the target, the fix is architectural (more frequent base backups, warm standby, smaller restore units via cells, 27) and not a more optimistic runbook.

A second region protects against a regional failure only if nothing on the recovery path still depends on the first one, and it is not cheap: AWS’s own guidance says a hot standby roughly doubles the cost and that most availability and RTO/RPO goals can be met inside one region. When the goal really needs two, the first design question is not “active/active or active/passive” but for each dataset, which region may accept writes right now, and how every other region knows it. Replication lag, conflicts and split brain all follow from that answer.

Failover is a split-brain problem. Promoting the standby while the old primary may still be alive and reachable by some clients produces two writers. DNS does not fence: answers are cached, and a client that already resolved the old address keeps using it. The promoted side needs an epoch that the storage itself checks, or the old primary must be made unable to accept writes before promotion (66). In an active/active table the equivalent trap is quieter: DynamoDB global tables in the default eventual mode resolve concurrent updates of one item by the latest timestamp, and their transactions are atomic only inside the region that ran them — a cross-row invariant from 90 does not survive the trip.

The recovery path must not live in the failed region. AWS’s example: the Route 53 control plane is hosted in us-east-1, so a failover that needs a DNS record change is a poor mechanism when the primary is us-east-1. Prefer pre-provisioned capacity and switches that need no control-plane change on the recovery path (101); keep certificates, keys, secrets and parameters region-local, with staggered expiry so one renewal failure cannot hit both; forbid cross-region calls inside a user journey and fail the whole journey over together. Deploy one region at a time, so a bad release cannot take both (27, 72).

Decision rule. Start from per-workload RTO and RPO (97). If waiting out a regional outage meets them, stay in one region with backups in another. Otherwise pick the simplest row of the table that meets the RPO, write down who owns each dataset’s writes and how ownership moves, monitor replication lag as an SLI, and keep the failover automated but the decision human. A failover that has not been run — both ways, with the same runbook as in a real event — is a hope, not a capability.

Eight fallacies (Deutsch / Gosling): the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology doesn’t change, there is one administrator, transport cost is zero, the network is homogeneous. Each is “obviously false” — and each is routinely baked into designs implicitly.

The list is folklore from Sun Microsystems in the 1990s: it is attributed to L. Peter Deutsch and colleagues there, with the eighth added later by James Gosling. It was never a paper with data; it is a summary of how distributed designs kept failing. The empirical record came later. Bailis and Kingsbury, “The Network is Reliable” (ACM Queue, 2014), collected public postmortems and studies of real partitions — switch failures, misconfigured firewalls, GC pauses that looked like partitions to peers — in systems whose designers had assumed the network away.

How they appear in backend code. Reliable network: an HTTP client with no timeout, so one slow dependency exhausts the caller’s threads. Zero latency: an ORM loop issuing one query per row, fine on a laptop and 400 round trips across zones in production. Infinite bandwidth: a service returning a full object graph when the caller needs two fields. Secure network: plaintext between services “because it is internal” (32). Topology doesn’t change: IPs cached past a pod reschedule. Zero transport cost: cross-region replication priced only after the bill (78).

A checkable form. Every service-to-service call in the code must answer three questions: timeout? retry policy (24), and is the operation safe to retry (22)? what happens when the other side is unavailable? No answer = fallacy #1 baked in. The opposite error is treating an in-process call as remote “to be safe” — that is accidental complexity (40). Not expressible in code; expressed as a review checklist for every network boundary.

CAP in its popular form is degenerate: P is not a choice — partitions happen, and the only choice is between C and A during a partition. PACELC is more honest: even without a partition (E), a choice between latency (L) and consistency (C) always exists — synchronous replication costs an RTT on every write.

CAP began as Eric Brewer’s conjecture in his PODC 2000 keynote and was proved by Seth Gilbert and Nancy Lynch (SIGACT News, 2002) for a precise model: C there means linearizability, A means every request to a non-failed node gets a response, and the proof is about behaviour while messages are lost. Brewer himself revisited it in “CAP Twelve Years Later: How the ‘Rules’ Have Changed” (IEEE Computer, 2012), stressing that partitions are rare and the choice is made per operation. PACELC is Daniel Abadi’s, from “Consistency Tradeoffs in Modern Distributed Database System Design” (IEEE Computer, 2012): if there is a Partition, choose A or C; Else, choose Latency or Consistency. In his classification the default configurations of Dynamo-style stores are PA/EL, and fully ACID systems are PC/EC.

Honest assessment. “CP or AP” as a database label is a marketing taxonomy; real systems are tuned per operation (quorum vs one in Cassandra, synchronous_commit in Postgres). The principle: the consistency level is a property of the operation, not of the system, and it is chosen by the invariant the operation protects.

PACELC does not invalidate CAP, it extends it: CAP describes behavior during a partition, PACELC adds the choice in normal operation. The “insufficient” in the title refers to the popular use of CAP as a label for databases. And synchronous_commit in PostgreSQL is a commit-acknowledgment and durability/replication setting, not a per-operation consistency-model switch.

The EL/EC choice in numbers. A PostgreSQL primary with a synchronous standby in another availability zone waits for the standby on every commit: about a millisecond or two inside a region, tens of milliseconds to another region. Asynchronous replication removes that wait and opens a window in which an acknowledged write exists on one machine only. Neither is “the right setting”; the invariant decides (65, 90) — a payment ledger pays the RTT, a view counter does not.

“Read your writes” is not a property of the database but a guarantee the service either gives or doesn’t, and it must be written into the contract. Consistency is not a single “stronger/weaker” scale but a set of observable guarantees: operation order, visibility of other clients’ writes, freshness, behavior within a session, binding to real time. Linearizability is the most expensive because it demands the last one: an operation must appear to take effect at a single instant between invocation and response. Most features are well served by session guarantees (read-your-writes, monotonic reads) — paying for linearizability by default means paying for something you don’t use.

// read-your-writes via a version token, no sticky sessions
record WriteResult(OrderId id, long lsn) {}          // the write's LSN/HLC goes back to the client

Order read(OrderId id, long minLsn) {
    var replica = pool.any();
    replica.waitForLsn(minLsn, TIMEOUT);             // replica catches up, or we route to primary
    return replica.get(id);
}
// client carries lsn in a cookie/header → its own writes are visible,
// others' only eventually; that is a session guarantee, declared explicitly

// same at the contract level: the request/response type carries the guarantee
type ReadConsistency int

const (
    Eventual ReadConsistency = iota
    ReadYourWrites
    Linearizable // documented as expensive: goes through the leader
)

func (s *Store) Get(ctx context.Context, id OrderID, c ReadConsistency) (Order, error)

A design-review test question: “which anomaly will the user pay for this optimization with — and do they know about it from the contract?”

Exposing long lsn leaks PostgreSQL into the contract. What goes out is an opaque consistency token with documented semantics, a wait timeout and defined behavior when it expires (fall back to reading from the leader or an explicit error).

No TTL-based lock (Redis, ZooKeeper lease, k8s lease) protects anything on its own: the holder can stall (a GC pause, a network delay), the TTL expires, someone else acquires the lock — and the “dead” holder wakes up and writes over them. The only correct scheme: the lock hands out a monotonic token, and the storage rejects writes carrying a stale token. The protection lives on the resource side, not the lock side — this is the E2E argument (21) applied to locking.

token, err := locker.Acquire(ctx, "settlement-2026-08-04") // token = 34
if err != nil { return err }
defer locker.Release(token)

// the store checks: accepted max token = 33 → 34 passes;
// a zombie arriving with token = 33 after 34 was seen is rejected
if err := store.Write(ctx, key, val, WithFenceToken(token)); err != nil { ... }

// the token is in the type: a write can't be built without it
pub struct Fenced<T> { token: FenceToken, payload: T }

impl Store {
    pub fn write(&self, w: Fenced<Write>) -> Result<(), Stale> { ... }
}

An honest assessment. The requirement on the resource is more precisely stated as a monotonic check: a fencing token is one special case, and a version or a conditional write / compare-and-swap expresses the same protection. If the resource supports none of these (S3 without a precondition, an external API), the lock buys you efficiency — not doing the work twice — but not correctness; confusing these two uses is the other half of “distributed lock” incidents, including the well-known Kleppmann vs antirez debate over Redlock. And separately: consensus (Raft/Paxos) is an algorithm, not a heuristic; you do not write your own implementation, you take an existing one.

NTP drift, leap smearing, VM pauses: System.currentTimeMillis() on two machines is not comparable, and “last write wins by timestamp” silently loses writes. Ordering is done with Lamport clocks / HLC / version vectors; wall clock is only for humans (logs, metrics) and for TTLs with a margin.

Three clocks that get confused. Wall clock (currentTimeMillis, CLOCK_REALTIME) answers “when, by the calendar”: logs, domain dates, TTLs with a margin. It jumps — forward and backward — on NTP correction. Monotonic clock (nanoTime, CLOCK_MONOTONIC) doesn’t jump and is therefore the only one fit for durations, timeouts and deadlines — but it is not comparable across machines and resets on restart, so storing it is pointless. Logical clocks (Lamport, vector, HLC) are the only source of the “happened before” relation between nodes. The typical bugs are exactly these substitutions: a duration computed from the wall clock goes negative during a correction, and “ordering” by the monotonic clock is meaningless outside the process. A separate layer is the fencing token (66): there you need not the order of events but the owner’s generation number.

# physical time + logical counter, monotonicity guaranteed
@dataclass(frozen=True, order=True)
class HLC:
    wall_ms: int
    logical: int

    def tick(self, now_ms: int) -> "HLC":
        if now_ms > self.wall_ms:
            return HLC(now_ms, 0)
        return HLC(self.wall_ms, self.logical + 1)   # clock went backwards: bump the counter

    def recv(self, other: "HLC", now_ms: int) -> "HLC":
        m = max(self.wall_ms, other.wall_ms, now_ms)
        l = max(self.logical if m == self.wall_ms else -1,
                other.logical if m == other.wall_ms else -1) + 1
        return HLC(m, l)

// review smell: comparing times recorded by different nodes
if (eventA.timestamp().isBefore(eventB.timestamp())) { ... } // ← flag it if A and B
                                                             //   were written by different nodes

Google TrueTime is the exception that proves the rule: Spanner buys clock comparability with atomic clocks and an explicit uncertainty interval (commit-wait). If you don’t have atomic clocks, you don’t have comparable timestamps.

Don’t mix the three clocks: wall clock for calendar time and display; monotonic clock for durations, timeouts and deadlines; HLC or version vectors for causal order. And last-write-wins is not always a bug: it is a legitimate convergence policy if losing a concurrent update is a deliberate decision rather than a surprise.

“Wrote to the database and published to Kafka” is two systems with no shared transaction: you can always crash between them, and one side will lie. There are two correct schemes: outbox (the event is written in the same DB transaction, a relay / CDC delivers it) or CDC directly from the WAL. “Publish first, then commit” and the reverse are both broken — they just produce phantoms of opposite sign.

-- one transaction: the business fact and the event are atomic
BEGIN;
INSERT INTO payments (id, amount_cents, status) VALUES ($1, $2, 'captured');
INSERT INTO outbox (event_id, aggregate_id, type, payload)
VALUES (gen_random_uuid(), $1, 'PaymentCaptured', $3);
COMMIT;

-- relay (Debezium / poller) reads outbox → Kafka; delivery is at-least-once,
-- so consumer-side dedup by event_id is mandatory (see 21)

// at-least-once loop: mark only after a confirmed publish
rows := tx.Query(`SELECT event_id, payload FROM outbox
                  WHERE published_at IS NULL ORDER BY seq LIMIT 100 FOR UPDATE SKIP LOCKED`)

for _, e := range rows {
    if err := producer.Send(ctx, e); err != nil { break } // not marked — we'll come back
    tx.Exec(`UPDATE outbox SET published_at = now() WHERE event_id = $1`, e.ID)
}

Outbox does not give exactly-once — it gives atomic writes and at-least-once delivery; deduplication still happens at the end (21). Listen-to-yourself and “just retry the publish” are not equivalents but well-known ways to get out of sync.

The relay loop is not idempotent by itself — it is at-least-once: the mark is set only after a confirmed publish, so a crash between send and update yields redelivery, not loss. Hence the mandatory details: claiming rows with FOR UPDATE SKIP LOCKED, a bounded batch, a publish timeout, and deduplication at the consumer (21).

Distributed ACID transactions exist (XA, 2PC) and give a stronger contract, atomicity across all participants at once, but they pay in coordination: participants hold locks until the protocol completes, and between prepare and the decision they are not free to decide on their own. A replicated coordinator (a decision log under Raft) removes the single point of failure, but it removes neither the locks nor the extra round trips. For long-running cross-service processes a saga is more practical: a chain of local transactions, each with a compensating action. The main cost people forget: isolation is gone. Intermediate states are visible to other participants, and a compensation does not “roll back”; it writes a new fact (a refund ≠ cancelling the charge).

// steps and compensations are declared in pairs; state is an explicit persisted type
sealed interface SagaStep permits ReserveInventory, ChargeCard, BookShipment {}

record ChargeCard(Cents amount) implements SagaStep {
    Compensation compensation() { return new RefundCard(amount); } // a new fact, not a rollback
}

// isolation by hand: a semantic lock; an order in status PENDING_PAYMENT
// is invisible to resale until the saga completes or is compensated

An honest assessment. As the number of steps grows, an explicit orchestrator (a state machine) is usually easier to observe and recover than implicit choreography: the question “who owns the process right now?” stops having a clear answer exactly when you need one at 3 a.m. This is a heuristic, not a threshold: for two participants with a clear contract, choreography is cheaper. A long-running saga = a workflow engine (Temporal and the like), which is not a new idea but a persisted saga orchestrator.

A cache is not a “speedup” but a deliberate weakening of consistency: the TTL is an SLA on staleness, and it must be derived from a business requirement (“the price may be wrong for 60 seconds”), not from “eh, let's say 5 minutes.” Event-driven invalidation with a TTL as a safety floor is a good default precisely as a pair: invalidations get lost not because of delivery semantics but because of operational failures (a crashed relay, an overflow, a routing error, a desync after a rebalance), and the TTL bounds the damage of such a loss. This is a default, not an obligation: durable write-through with explicitly stated guarantees is also correct.

The best-documented account of this at scale is Nishtala et al., “Scaling Memcache at Facebook” (NSDI 2013). Two of its mechanisms answer the failures above: invalidations are driven from the database’s commit log rather than from application code, so a crashed web server cannot skip one, and leases stop a slow reader from writing a stale value back after a newer write deleted the key — the race every cache-aside implementation has by default. HTTP made staleness explicit in the protocol: Cache-Control: max-age and stale-while-revalidate (RFC 5861, on top of the caching semantics of RFC 9111) state how old a response may be and for how long it may be served while refreshing.

// TTL as a contract, not a constant buried deep in the code
type CachePolicy struct {
    TTL                  time.Duration // = the documented staleness budget
    StaleWhileRevalidate time.Duration // serve stale, refresh in the background
}

var pricePolicy = CachePolicy{TTL: 60 * time.Second, StaleWhileRevalidate: 10 * time.Second}
// review question: "why 60?" must have an answer citing the requirement

Cache-aside / read-through / write-through / write-behind is not a matter of taste: write-behind loses data when the cache goes down and is acceptable only for recoverable data (counters, metrics). A key with two writers (the service and the invalidator) is a race by construction; version the value or write through a single point.

Where no staleness budget exists. Some reads must not be stale at all: a balance checked before a withdrawal, a permission just revoked, an idempotency key (22). There the cache either is bypassed for that read or becomes part of the write path with the same guarantees as the database. If nobody can state the budget in business terms, that is the finding: the cache is hiding a consistency decision nobody made. Stampedes on expiry are a separate problem (71).

Expiry of a hot key = a synchronized volley from all its subscribers into the origin, a self-DDoS on a schedule. Three mandatory mechanisms: collapse concurrent misses into one request (singleflight), randomize TTLs (so keys don't expire in unison), and cache absence (otherwise every request for a nonexistent id hits the database).

var g singleflight.Group

func (s *Svc) Product(ctx context.Context, who Principal, id string) (Product, error) {
    // one key for cache AND singleflight: tenant + access scope + id,
    // or a caller joins a flight that loaded someone else's view
    key := who.TenantID + "|" + who.Scope + "|" + id
    if p, ok := s.cache.Get(key); ok { return p, nil }
    v, err, _ := g.Do(key, func() (any, error) {      // N misses → 1 origin request
        p, err := s.db.Product(ctx, who, id)
        if errors.Is(err, ErrNotFound) {
            s.cache.SetNegative(key, 30*time.Second)  // negative cache, short TTL
            return Product{}, err
        }
        s.cache.Set(key, p, jitterTTL(5*time.Minute)) // ±20% — desynchronizes expiry
        return p, err
    })
    ...
}

# probabilistic early refresh: hot keys are refreshed BEFORE they expire,
# with probability rising as the TTL approaches
import math, random, time

def should_refresh(expires_at: float, delta: float, beta: float = 1.0) -> bool:
    return time.time() - delta * beta * math.log(random.random()) >= expires_at

Negative caching without a short TTL is a way to cache a transient error forever; cache absence, not errors (an origin 5xx is never written to the cache).

Limits of applicability. Negative caching is needed where misses are genuinely expensive; otherwise it is an extra layer. The cache key must include the tenant and access rights — otherwise singleflight will merge requests from users with different visibility and hand out someone else's data. And cancelling the first caller's context must not cancel the work for everyone who joined it.

An isolation level is a list of anomalies you agree to handle yourself. Snapshot isolation (PostgreSQL’s REPEATABLE READ) stops lost updates on the same row, but not write skew: two transactions read an overlapping set, each checks the invariant against its own snapshot, then each writes a different row — no write-write conflict, both commit, and the invariant that spans the rows is gone. So the question at review is not “which level do we use” but “which invariant does this transaction protect, and is it a single-row fact, a multi-row fact, or a predicate over rows that may not exist yet”.

-- Invariant: shift 1 always has at least one doctor on call.
CREATE TABLE on_call (doctor text PRIMARY KEY, shift_id int NOT NULL, on_call bool NOT NULL);
INSERT INTO on_call VALUES ('alice', 1, true), ('bob', 1, true);

-- Sessions A and B interleave; each step runs in time order.
BEGIN ISOLATION LEVEL REPEATABLE READ;                             -- A
BEGIN ISOLATION LEVEL REPEATABLE READ;                             -- B
SELECT count(*) FROM on_call WHERE shift_id = 1 AND on_call;       -- A: 2, "safe to leave"
SELECT count(*) FROM on_call WHERE shift_id = 1 AND on_call;       -- B: 2, "safe to leave"
UPDATE on_call SET on_call = false WHERE doctor = 'alice';         -- A
UPDATE on_call SET on_call = false WHERE doctor = 'bob';           -- B: different row, no wait
COMMIT;                                                            -- A: ok
COMMIT;                                                            -- B: ok
-- Result: nobody on call. Both transactions were correct in isolation.

-- Same interleaving with BEGIN ISOLATION LEVEL SERIALIZABLE:
-- A commits; B's COMMIT fails and bob stays on call:
--   ERROR: could not serialize access due to read/write dependencies among transactions
--   SQLSTATE 40001, "Canceled on identification as a pivot, during commit attempt."
-- The caller re-runs B from BEGIN, including the SELECT: the retry now sees count = 1.

-- Alternative at READ COMMITTED: lock the rows the decision reads.
SELECT doctor FROM on_call WHERE shift_id = 1 AND on_call FOR UPDATE;
-- B blocks until A commits, then gets only 'bob' back and must refuse.
-- At REPEATABLE READ the same statement fails B with 40001 instead.
-- Neither variant stops a concurrent INSERT of a new matching row (a phantom).

Decision rule. A single-row invariant is covered by READ COMMITTED plus an atomic statement (UPDATE … SET x = x - 1 WHERE x > 0) or a version check (66). An invariant expressible as a constraint belongs in the schema: UNIQUE, CHECK, EXCLUDE — the database enforces it at every level. A multi-row invariant over rows that already exist can be protected by SELECT … FOR UPDATE on exactly those rows; the lock must cover everything the decision read, and it blocks. A predicate over rows that may be inserted concurrently (“no overlapping booking”, “at most N active sessions”) needs SERIALIZABLE, a materialized conflict row that every writer updates, or a coarser lock. SERIALIZABLE is the only option that stays correct when a new transaction type appears that nobody reviewed against the old ones.

What SERIALIZABLE costs in PostgreSQL. It is SSI: snapshot isolation plus tracking of read/write dependencies through non-blocking SIReadLock predicate locks; when two consecutive dependencies between concurrent transactions appear, one of them is aborted (the pivot, if it has not committed yet). The detection is conservative, so some aborted transactions would have been serializable. The price is paid in retries, and the retry contract is strict: SQLSTATE 40001 can arrive on any statement including COMMIT; the whole transaction, including the reads and the logic that chose what to write, is re-run (any side effect outside the database inside that code needs 22, and the loop needs a bound, 24); nothing read inside it counts as true until it commits. Sequential scans take relation-level predicate locks and raise the failure rate, so the queries need indexes. Read-only reports can use SERIALIZABLE READ ONLY DEFERRABLE, which waits for a safe snapshot and then cannot be rolled back by a serialization failure.

Level names are not portable. ANSI defines levels by phenomena, and Berenson et al. showed those definitions are ambiguous and that snapshot isolation is incomparable with locking REPEATABLE READ: each allows an anomaly the other forbids. The same keyword means different guarantees in different engines, and read replicas or caches in front of the database change them again (65, 70). Specify the anomaly, not the keyword.

Replication copies bytes; consensus decides which command is entry N, once, for every replica. Deterministic state machines that apply the same log in the same order end in the same state — that is the whole product. In Raft an entry is committed when the leader that created it has it on stable storage of a majority; only then is it applied and acknowledged. Every message carries a term (an epoch): a server that sees a higher term steps down, and a request with a stale term is rejected. Safety — no two replicas apply different commands at the same index — must never depend on timing; only liveness does.

// Default Get is linearizable: it goes through Raft and reflects every acknowledged write.
cur, err := cli.Get(ctx, "config/limits")
if err != nil {
    return err
}
// Serializable: answered by the local member, may lag the quorum.
if _, err := cli.Get(ctx, "config/limits", clientv3.WithSerializable()); err != nil {
    return err
}

rev := int64(0) // a missing key compares as ModRevision 0
if len(cur.Kvs) == 1 {
    rev = cur.Kvs[0].ModRevision
}
// Compare-and-swap on the revision we read; the replicated log orders the contenders.
txn, err := cli.Txn(ctx).
    If(clientv3.Compare(clientv3.ModRevision("config/limits"), "=", rev)).
    Then(clientv3.OpPut("config/limits", next)).
    Commit()
if err != nil {
    return err // outcome unknown: the write may have committed; re-read before retrying
}
if !txn.Succeeded {
    return errLostRace // someone committed in between: re-read, recompute, retry
}

Where the theorem ends. FLP proves that no deterministic protocol guarantees agreement and termination in a fully asynchronous system if even one process may crash. Paxos and Raft answer by never giving up safety and buying termination with timing assumptions: Raft keeps a stable leader while broadcastTime ≪ electionTimeout ≪ MTBF, and randomized election timeouts make split votes unlikely, not impossible. The engineering consequences: a majority of 2f+1 servers tolerates f failures, so a fourth server adds cost and no tolerance; during an election or a partition the minority side must refuse writes — the C in 64; and every persisted term, vote and log entry is fsynced before the reply, or a restart can vote twice.

The three places implementations get it wrong. Reads: a leader answering from local state may already be deposed. Raft’s linearizable read commits a no-op at the start of the term and confirms leadership with a majority heartbeat before answering; a leader lease skips the round trip but makes safety depend on bounded clock drift (67) — the one place the timing rule is knowingly bent. Membership: changing the voter set must keep old and new majorities overlapping (joint consensus, or one server at a time); the single-server scheme published in 2014 had a split-brain bug fixed by one rule — a leader appends a configuration entry only after committing an entry from its own term. Clients: a retry after a timeout can apply a command twice; linearizable semantics need client ids and sequence numbers deduplicated inside the state machine (22).

Consensus orders commands inside the group. A lock or leadership taken from it protects an external resource only with a fencing token checked by that resource (66); an etcd revision like the one above can serve as that token. Use an existing implementation (etcd, ZooKeeper, a database’s built-in quorum) and test it under faults rather than trusting the paper’s proof for your code (98, 99).