Using KRaft Consensus Protocol
1. Understanding KRaft Architecture
| Concept | Description | Detail |
|---|---|---|
| KRaft | Self-managed metadata via Raft | No ZooKeeper |
| Controllers | Raft quorum storing metadata | Odd count |
| Active Controller | Raft leader handling writes | One at a time |
2. Understanding Metadata Replication
| Concept | Description | Detail |
|---|---|---|
| Metadata Log | __cluster_metadata topic | Event-sourced |
| Raft Replication | Quorum-replicated records | Majority ack |
| Brokers Subscribe | Replay metadata log | Eventually consistent |
Example: Inspect metadata log
kafka-metadata-shell.sh --snapshot \
/var/lib/kafka/__cluster_metadata-0/00000000000000000000.log3. Configuring Metadata Log
| Config | Description | Default |
|---|---|---|
| metadata.log.dir | Metadata log location | First log.dir |
| metadata.log. max.record.bytes |
Max metadata record | — |
| metadata.max. retention.ms |
Snapshot/retention | — |
4. Setting Controller Quorum Voters
| Config | Format | Detail |
|---|---|---|
| controller.quorum.voters | id@host:port | Comma-separated |
| Count | 3 or 5 voters | Odd for quorum |
| Fault Tolerance | N voters tolerate (N-1)/2 | Majority |
5. Formatting Storage
| Step | Command | Detail |
|---|---|---|
| random-uuid | Generate cluster ID | Once |
| format | Init log dirs | Per node |
| --ignore-formatted | Skip if done | Idempotent |
Example: Format storage
bin/kafka-storage.sh format -t $CLUSTER_ID \
-c config/kraft/server.properties6. Starting KRaft Controller
| Aspect | Description | Detail |
|---|---|---|
| process.roles | controller | Dedicated mode |
| controller. listener.names |
CONTROLLER | Required |
| Start | kafka-server-start.sh | Controller config |
7. Starting KRaft Broker
| Aspect | Description | Detail |
|---|---|---|
| process.roles | broker | Data node |
| controller. quorum.voters |
Points to controllers | Required |
| Start | broker.properties | Connects to quorum |
8. Using Combined Mode
| Aspect | Description | Detail |
|---|---|---|
| Combined | broker,controller in one process | Dev/small |
| Pros | Fewer nodes | Simpler |
| Cons | Not for large prod | Isolation lost |
9. Configuring Controller Listener
| Config | Description | Detail |
|---|---|---|
| controller. listener.names |
Listener for controller traffic | CONTROLLER |
| listeners | Must include controller port | :9093 |
| Security | Map to SSL/SASL if needed | protocol.map |
Example: Controller listener
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER10. Migrating from ZooKeeper
| Step | Action | Detail |
|---|---|---|
| Provision | Deploy KRaft controllers | Migration mode |
| Enable Migration | zookeeper.metadata.migration.enable | Dual-write |
| Finalize | Remove ZK config | KRaft-only |
Warning: Kafka 4.0 removes ZooKeeper; migrate to KRaft on 3.x before upgrading to 4.0.