The partition key is semantics, not plumbing — it fixes order, locality and the unit of skew

heuristic
established
fragment

Scope. The remapping bounds are theorems about the hash functions in their papers’ models (random hash functions, even division of objects); they say nothing about ordering, in-flight data or skewed key popularity, which is where the card’s engineering advice comes from. The Kafka and librdkafka behaviour described is that of their current default partitioners; a custom partitioner changes all of it.

Versions. Kafka 4.3 Java producer (BuiltInPartitioner: murmur2 of the serialized key; sticky partitioning for null keys); librdkafka master CONFIGURATION.md (default partitioner consistent_random); Flink 2.x (StreamExecutionEnvironment.setMaxParallelism, key groups up to 32768).

Conflicts with: Even load ↔ per-key order

Primary sources: Apache Kafka 4.3 — Design: the producer, semantic partitioning, Apache Kafka 4.3 — Introduction: events with the same key go to the same partition and are read in write order

References: Apache Kafka 4.3 — Basic operations: modifying topics (effects of adding partitions), Apache Kafka 4.3 — Producer configs (partitioner.class, partitioner.ignore.keys), Apache Kafka 4.3 — BuiltInPartitioner.partitionForKey (murmur2), librdkafka — CONFIGURATION.md (partitioner), Apache Flink 2.3 — Production readiness checklist (maximum parallelism, operator uids), Apache Flink 2.3 — Parallel execution (maximum parallelism and key groups)

Other claims on this card, typed separately:

- theorem
- established
Consistent hashing (Karger et al.) gives a monotone, balanced ranged hash family: when buckets are added, an item moves only from an old bucket to a new one, never between old buckets; balance uses O(log C) points per bucket.
Primary sources: David Karger, Eric Lehman, Tom Leighton, Matthew Levine, Daniel Lewin, Rina Panigrahy — Consistent Hashing and Random Trees (STOC 1997)
- theorem
- established
For any mapping that divides objects evenly among n servers, at least 1/n of the objects must be remapped when a server comes up or goes down; highest-random-weight (rendezvous) mapping attains this minimum.
Primary sources: David G. Thaler, Chinya V. Ravishankar — Using Name-Based Mappings to Increase Hit Rates (IEEE/ACM ToN, 1998)
- theorem
- established
Jump consistent hash needs no stored state and keeps, in expectation, n/(n+1) of the keys in place when growing from n to n+1 buckets, but requires buckets to be numbered sequentially.
Primary sources: John Lamping, Eric Veach — A Fast, Minimal Memory, Consistent Hash Algorithm (2014)

Choosing a partition key decides three things at once: what is ordered (Kafka orders records only within a partition), what is co-located (every keyed join, aggregate and piece of state for one key lives on one partition — 82), and what cannot be split (a key whose traffic exceeds one partition’s capacity saturates it no matter how large the cluster is). The partition count is part of the same decision and a one-way door (20): with hash(key) % N, changing N reroutes most keys.

What changing the count does. The Java producer maps a keyed record to murmur2(serialized key) % partitions; keyless records go to a sticky partition that switches after roughly batch.size bytes. After partitions are added, a key may land on a different partition than its history, Kafka does not move existing data, consumers with auto.offset.reset=latest can skip records produced to the new partitions before they notice them, and the count can never be reduced. Per-key order across the change is therefore gone unless the consumer drains the old placement first. The hash is also a cross-client contract: librdkafka’s default partitioner is CRC32-based, so a librdkafka-based producer (confluent-kafka-python, confluent-kafka-go) on defaults and a Java producer send the same key to different partitions unless librdkafka is set to murmur2_random. Flink hashes a key into one of maxParallelism key groups and assigns key-group ranges to subtasks; rescaling moves whole groups, but maxParallelism itself cannot change without discarding keyed state.

Remapping is a solved problem; order and hot keys are not. Consistent hashing (Karger et al.) is monotone — a new bucket takes items only from old buckets, never shuffles them between old ones — and needs virtual points per bucket for balance. Rendezvous (HRW) hashing reaches the minimum possible disruption, 1/n of the objects when a server comes or goes; jump hash needs no state but only works with sequentially numbered buckets. All three bound how much moves, none of them preserves order for what does move, so a keyed topic is resized by expand — migrate — contract into a new topic (17) rather than by --partitions. A hot key is a data-model problem: split it by salting only when every downstream operation is commutative and is re-merged by the unsalted key, or route it to its own topic; otherwise it sets the tail for everyone sharing its partition (26, 93, 103).

fragment

Java · Kafka clients 4.x · Flink 2.x

// The key is the contract: all events of one account share a partition, in send order.
producer.send(new ProducerRecord<>("payments", event.accountId(), event));

// Anti-pattern for this topic: a null key spreads load evenly (sticky partitioner)
// and destroys per-account order.
// producer.send(new ProducerRecord<>("payments", null, event));

// A key hotter than one partition is not fixed by adding partitions. Salting trades
// per-key order for spread; valid only because this topic feeds commutative sums
// that are re-aggregated by accountId downstream.
String key = hotAccounts.contains(event.accountId())
    ? event.accountId() + "#" + ThreadLocalRandom.current().nextInt(SALT_BUCKETS)
    : event.accountId();
producer.send(new ProducerRecord<>("payment-totals", key, event));

// Flink: key -> one of maxParallelism key groups -> a subtask's range of groups.
// Rescaling moves whole groups; changing this number later discards keyed state.
env.setMaxParallelism(720);