#kraft
Posts
-
Digging into Kafka's Source 2: The Big Picture — Clients, Brokers, Controllers and Partitions
A big-picture map for intermediate developers opening the Kafka source for the first time. Starting from the main method of kafka.Kafka that kafka-server-start.sh calls, we follow how process.roles decides whether BrokerServer, ControllerServer and SharedServer get built, what a broker creates when it starts, how the controller decides topics, partitions and leaders through the metadata log and spreads them to the brokers, how a client finds a leader through a Metadata response, and which way the modules depend on each other — all checked against the real source and examples I actually ran — and we mark which part of the map Parts 3–12 zoom into.
-
Digging into Kafka's Source 10: KRaft and Metadata — How the Controller Remembers the Cluster
The cluster's ledger — topics, partitions, leaders, the list of brokers — is stored as records in __cluster_metadata, an ordinary Kafka log, and the controllers keep that log together with Raft. We follow it through the source: the roles in KafkaRaftClient (Prospective, Candidate, Leader, Follower) with Vote and Fetch, a high watermark set by a majority of voters, a QuorumController that runs on a single event thread and applies records to memory before they commit (and rolls them back with SnapshotRegistry), the MetadataLoader that turns records into MetadataDelta and MetadataImage on the broker, and heartbeats and fencing — checked with kafka-dump-log, kafka-metadata-quorum and a three-controller election.