Eval-driven development

Evals are tests for a non-deterministic component. Without evals, swapping a prompt/model is flying blind. The hierarchy by cost: assertion-based → golden set + metrics → LLM-as-judge (which itself needs calibration against human labels).

There is no single inventor; the practice grew out of NLP benchmarking. HELM (Liang et al., Stanford, 2022) argued that a model must be evaluated on many scenarios and metrics at once, not one score, and OpenAI’s open-source Evals framework (2023) made per-application eval suites a normal engineering artifact. The case for treating judges with suspicion is measured: Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (NeurIPS 2023), found that strong models agree with human preferences at roughly the rate humans agree with each other, and also documented position bias, verbosity bias and a preference for their own answers.

# Level 1: cheap programmatic checks, run on every prompt PR
def eval_extraction(case: Case) -> Score:
    out = pipeline(case.document)
    return Score(
        json_valid   = is_valid_json(out),
        schema_ok    = validate_schema(out, InvoiceSchema),
        amounts_ok   = out["total"] == case.expected_total,   # exact match, not similarity
        no_leak      = case.ssn not in out_raw,
    )

# Level 2: golden set is versioned next to the prompt; regression = red CI
# Level 3: LLM-judge — only for style/tone,
#            periodically checked against humans

Level 1 is not the whole eval. A non-deterministic component requires: repeated runs and confidence intervals instead of a single number; slices (by language, length, document type), because the average hides a failure on a slice; adversarial cases; calibration of the LLM judge against human labels; a check for golden-set leakage into the prompt; and monitoring for drift in production. Cost and latency are measured alongside quality, not separately.

How big a difference is real. On 100 golden cases a 90 % pass rate has a 95 % binomial interval of roughly ±6 points, so a prompt change that moves it from 90 to 93 is indistinguishable from noise — and a non-deterministic model adds run-to-run variance on top. Before comparing two versions, decide the minimum difference that matters and size the set (or the number of repeats) to detect it; compare paired, on the same cases.

Limits. An eval measures only what is in the set, and a set tuned against for months becomes a target (Goodhart): the prompt fits the golden cases and production drifts away from them. Keep a held-out slice nobody optimizes on, and refresh the set from real failures. Evals do not replace deterministic guards (48, 53): a 99 % score on “no PII leaked” is a 1 % leak rate.