Correctness can only be ensured at the endpoints; intermediate layers provide optimization, not a guarantee. Corollary: a broker's exactly-once ends at the broker's boundary — beyond it, deduplication must live in the consumer.
The precise statement. Kafka transactions provide true EOS in a read-process-write loop where input, output and offsets all live inside Kafka: the offset commit and the result write are atomic. The guarantee disappears exactly where an external sink appears (a database, a payment gateway, an email): it shares no transaction with the broker. So “exactly-once” is not marketing but a scope of applicability, outside which only E2E deduplication works.
-- The dedup marker and the business-state change go in ONE transaction.
-- Otherwise two consumers pass the check at once, or the process crashes
-- between the effect and the marker, and the event is applied twice.
BEGIN;
-- The effect is conditional on the claim: a duplicate inserts no row,
-- so `claimed` is empty and the UPDATE touches 0 rows.
WITH claimed AS (
INSERT INTO consumed_events (event_id) VALUES (:event_id)
ON CONFLICT DO NOTHING
RETURNING event_id
)
UPDATE accounts SET balance = balance + :amount
WHERE id = :account_id AND EXISTS (SELECT 1 FROM claimed);
COMMIT;void handle(ConsumerRecord<String, OrderEvent> rec) {
// Dedup and effect inside one storage transaction, not as two
// calls around process(): the process can die in between.
tx.run(session -> {
if (!session.claimEvent(rec.value().eventId())) return; // INSERT .. ON CONFLICT
applyBusinessChange(session, rec.value());
});
// offset is committed AFTER the transaction: a retry is safe, a loss is not
consumer.commitSync();
}Where this stops working. The scheme above requires the effect and the marker to live in the same transactional resource. If the side effect is external — a payment gateway, an email, a call to someone else's API — there is no shared transaction, and three options remain: an idempotency key on the external service's side, an outbox (68) followed by reconciliation, or an explicit reconciliation procedure. Deduplication in a separate store with a TTL is an optimization (“don't do the work twice”), not a correctness guarantee.