Kafka Partition and Replication

Kafka Partition and Replication

Plan partitions, watch key routing, and test replication choices

animation
kafka
stream-processing
distributed-systems
Kafka Partition Planning Replication

Route keyed IoT events, then test the partition plan under load and failure.

Follow one message through key hashing, partition selection, leader append, replica sync, and consumer assignment. The same scene shows why hot keys, too few partitions, or weak ISR settings limit a Kafka design.

Key route sensor-17 -> P2
Hot partition P2 has 34% load
Consumer lanes 4 active, 0 idle
Write state Available

Partition assignment and replica placement

A keyed message keeps per-device order by landing in one partition. Replicas protect that partition only when enough in-sync copies remain.

Hash route
Producer sensor topic key sensor-17 Partitioner hash % 4 Topic partitions ordered logs, independent offsets Broker cluster RF=3, min ISR=2 Consumer group each partition has one consumer in this group event Current step Hash key to a partition. Per-key ordering follows this chosen log. Offsets are local to that partition. Producer sensor topic key sensor-17 Partitioner hash % 4 Topic partitions Broker cluster RF=3, min ISR=2 Consumer group

Controls

Scenario
View Focus

Readouts

Route sensor-17 hashes to partition 2

Messages with this key keep relative order in partition 2.

Consumer group 4 active consumers

Consumer group parallelism is capped by partition count.

Replication 3 copies across 3 brokers

The leader handles appends, and followers must stay in sync to be failover candidates.

Diagnosis Plan is balanced

Partition count, consumer count, and ISR settings are aligned for this scenario.

Model Reading

The selected key is routed with a teaching hash:

partition = stableHash(key) % partition_count

Kafka preserves order inside each partition. A topic with many partitions has many ordered logs, not one global order.

Accuracy Notes
  • Replication factor counts every copy: one leader plus follower replicas.
  • With unclean leader election disabled, a new leader should come from the ISR.
  • `acks=all` succeeds only when the ISR size satisfies `min.insync.replicas`.
  • Adding consumers beyond partition count creates idle consumers in the same group.
  • Changing partition count can change key routes unless a custom partitioning strategy is used.
Guided Checks
  1. Raise consumers above partitions and note where idle capacity appears.
  2. Increase key skew and watch one partition become the bottleneck.
  3. Fail the leader with a lagging follower and compare `acks=1` with `acks=all`.
  4. Raise partition count and observe that the selected key may move to a different partition.
TryLoad Balanced sensors with 4 Partitions and 4 Consumer instances, then press Play. Enable Make the selected leader fail and compare acks=all with acks=1.
ObserveThe selected key remains ordered in one partition, and active consumers cannot exceed partition count. With a lagging follower, the ISR readout determines whether an acknowledged write remains available.
ExplainStable hashing maps each key to a single ordered log. Replication creates failover copies, while producer acknowledgements and minimum ISR jointly determine when a write is accepted.
Technical boundaries. The planner abstracts broker disk throughput, ISR churn timing, compression and batching, rack-aware placement, consumer rebalance pauses, and exactly-once processing.