Design for replay: reprocessing is a routine operation, not a fire drill

practice
context-dependent
fragment

Conflicts with: Replay ↔ side effects, Retention as cost ↔ replay horizon, Short checkpoint interval ↔ checkpoint overhead and state size

A bug in a transformation, a new column, a corrupted interval: history will be replayed; the only question is whether that is designed in or improvised at 3 a.m. Requirements by construction: transformations are deterministic (no now(), no random(), no lookups into mutable unversioned reference data), sinks are idempotent or transactional, the position (offset / checkpoint) is a first-class managed artifact, and source retention ≥ the replay horizon.

fragment

Java

// replay's enemy: nondeterminism hidden inside a transformation

// BAD:    enrich(event, dict.current())        // a replay a month later gives
//                                              // a different result
// GOOD:   enrich(event, dict.asOf(event.ts())) // temporal join on event time:
//                                              // the result is reproducible
//                                              // within the data contract

// a replay request as an explicit type: bounds, rate, destination
type ReplayJob struct {
    Source    TopicRange   // {topic, partitions, fromOffset/fromTs, toTs}
    Transform VersionedRef // transformation sha: which code is replayed
    Sink      SinkRef      // the same prod topic OR a shadow one; an explicit choice
    RateLimit TokenBucket  // replay must not crowd out live traffic (bulkhead, 24)
}

Replaying into the same downstream requires the whole chain below to be idempotent (22, dedup by event_id: 21); otherwise, replay into a shadow sink, compare, cut over. Kappa architecture is the replay principle elevated to an architecture: batch = a replay of the stream. The consequence for retention: a topic's retention period is not a disk cost but the insurance horizon for reprocessing; cutting it is a one-way-door decision (20).

Byte-for-byte equality is achievable only if the entire transitive dependency is versioned: transformation code, schemas and parsers, reference data, libraries, the floating-point environment. That is why “reproducibility” is verified by comparison rather than declared, and is stated within the bounds of the data contract. This also covers the comparison strategy, reconciliation, retention limits imposed by law and privacy, and separating resources between replay and live traffic.