Kafka Partition and Replication

Plan partitions, watch key routing, and test replication choices

Kafka Partition Planning Replication

Route keyed IoT events, then test the partition plan under load and failure.

Follow one message through key hashing, partition selection, leader append, replica sync, and consumer assignment. The same scene shows why hot keys, too few partitions, or weak ISR settings limit a Kafka design.

Key route sensor-17 -> P2

Hot partition P2 has 34% load

Consumer lanes 4 active, 0 idle

Write state Available

Partition assignment and replica placement

A keyed message keeps per-device order by landing in one partition. Replicas protect that partition only when enough in-sync copies remain.

Hash route

Controls

Scenario

View Focus

Partitions 4 Consumer instances 4 Key skew 15%

Brokers 3 Replication factor 3 Minimum ISR 2 Producer acks acks=all acks=1 acks=0

Fail the target leaderElects another in-sync replica if one is available. Make one follower lagRemoves that follower from the ISR for the selected partition.

Readouts

Route sensor-17 hashes to partition 2

Messages with this key keep relative order in partition 2.

Consumer group 4 active consumers

Consumer group parallelism is capped by partition count.

Replication 3 copies across 3 brokers

The leader handles appends, and followers must stay in sync to be failover candidates.

Diagnosis Plan is balanced

Partition count, consumer count, and ISR settings are aligned for this scenario.

Model Reading

The selected key is routed with a teaching hash:

partition = stableHash(key) % partition_count

Kafka preserves order inside each partition. A topic with many partitions has many ordered logs, not one global order.

Accuracy Notes

• Replication factor counts every copy: one leader plus follower replicas.
• With unclean leader election disabled, a new leader should come from the ISR.
• `acks=all` succeeds only when the ISR size satisfies `min.insync.replicas`.
• Adding consumers beyond partition count creates idle consumers in the same group.
• Changing partition count can change key routes unless a custom partitioning strategy is used.

Guided Checks

1. Raise consumers above partitions and note where idle capacity appears.
2. Increase key skew and watch one partition become the bottleneck.
3. Fail the leader with a lagging follower and compare `acks=1` with `acks=all`.
4. Raise partition count and observe that the selected key may move to a different partition.