Kafka
Posts
-
Digging into Kafka's Source 1: How I Read It with AI, and the Map
The first post of a series that reads the Kafka source — it starts by showing what Kafka is, how it works and what it is for, with diagrams and a 3-minute hands-on run. It covers which version is read (trunk, 4.5.0-SNAPSHOT) and how it relates to the stable 4.4.0, how to get the same source and build it with Gradle in Docker, the processMessages step that generates code from JSON schemas, a module map of the roughly 720,000-line main codebase, the big road one record travels from send() through the disk and replication to poll(), and how AI-written notes are re-checked against the source.
-
Digging into Kafka's Source 2: The Big Picture — Clients, Brokers, Controllers and Partitions
A big-picture map for intermediate developers opening the Kafka source for the first time. Starting from the main method of kafka.Kafka that kafka-server-start.sh calls, we follow how process.roles decides whether BrokerServer, ControllerServer and SharedServer get built, what a broker creates when it starts, how the controller decides topics, partitions and leaders through the metadata log and spreads them to the brokers, how a client finds a leader through a Metadata response, and which way the modules depend on each other — all checked against the real source and examples I actually ran — and we mark which part of the map Parts 3–12 zoom into.
-
Digging into Kafka's Source 3: The Wire Protocol — Requests and Responses Generated from JSON Schemas
The bytes a producer and a broker exchange over TCP always have one shape, [4-byte length][request header][body]. We follow in the source how 203 JSON schemas that define that shape become 391 Java classes at build time, how the ApiVersions exchange picks a version, and how flexible versions and tagged fields let old and new versions live together, then check the bytes ourselves with an example that sends requests over a raw socket using nothing but the generated classes.
-
Digging into Kafka's Source 4: The Broker's Network Layer — Acceptors, Processors and Request Handlers
We follow, in SocketServer.scala, how the three kinds of threads in a broker's jstack — socket-acceptor, network-thread and request-handler — share the work of a single request. One Acceptor per listener; a Processor reads the bytes, puts the request on the request queue and mutes the connection; a request handler branches on KafkaApis.handle()'s match; and the response goes back to the Processor that read the request. With jstack, a raw-socket client and the request log we check how mute/unmute keeps one connection's requests in order by handling them one at a time, and where num.network.threads, num.io.threads and queued.max.requests take effect, and we compare it all with Tomcat's NIO connector.
-
Digging into Kafka's Source 5: Inside the Producer — From send() to Batches and the Sender Thread
KafkaProducer.send() sends nothing over the network. It serializes the record, picks a partition, appends it to a per-partition batch in the RecordAccumulator and hands back a Future; the one thread that actually sends is kafka-producer-network-thread. We follow in the source the if in ready() where linger.ms and batch.size apply, the built-in partitioner that sticks keyless records to one partition, the BufferPool that hands out batch memory and max.block.ms, retries and delivery.timeout.ms, and max.in.flight.requests.per.connection, then check them with examples that count batches with kafka-dump-log.sh and print which thread runs the callback.
-
Digging into Kafka's Source 6: How the Broker Accepts Records — The Record Batch Format and the Produce Path
A record batch sent by the producer is a 61-byte header followed by records squeezed with varints. We read those bytes straight out of a segment file, then follow one Produce request through KafkaApis → ReplicaManager → Partition → UnifiedLog → LogValidator → LogSegment with the source and the broker's TRACE log. Using hand-built Produce requests we confirm that the broker writes the batch almost as-is, fixing only a few header fields (the CRC stays the same), when it recompresses because the topic's compression.type differs, and where acks=0/1/all change the moment the response goes out.
-
Digging into Kafka's Source 7: Log Storage — Segments, Indexes, Retention and Compaction
When does a file like 00000000000000003830.log appear in a partition directory, how is one offset turned into a byte position in that file, who deletes a segment once retention.ms has passed, and how does cleanup.policy=compact sweep away old values of the same key? We follow UnifiedLog, LocalLog, LogSegment, OffsetIndex and LogCleaner in the source, and check with examples the five roll conditions, the sparse index that writes one entry per 4 KB and lives in mmap, the two-step delete through a .deleted rename, the SkimpyOffsetMap built from MD5 hashes of keys, and the recovery point used when a broker comes back from a crash.
-
Digging into Kafka's Source 8: The Read Path — Fetch Requests, Long Polling and Zero-Copy
We follow a consumer's Fetch request through the Kafka 4.5.0-SNAPSHOT broker — KafkaApis.handleFetchRequest() → ReplicaManager.fetchMessages() → UnifiedLog.read() → LogSegment.read() — until it becomes a slice of a file (FileRecords), how it waits in the purgatory (DelayedFetch) for up to fetch.max.wait.ms when there is not enough data, and how FileChannel.transferTo() finally sends it to the socket without passing through the JVM. Request logs, JMX and JFR examples show incremental fetch sessions, why consumers and followers share one Fetch, and why zero-copy disappears under TLS.
-
Digging into Kafka's Source 9: Replication — The ISR, the High Watermark and the Wait Behind acks=all
A follower broker's ReplicaFetcherThread sends Fetch requests to the leader and copies its log, and the leader learns each follower's position from the offset in those Fetch requests and raises the high watermark. We follow, in Partition.scala and AbstractFetcherThread.scala, the rule for dropping out of the ISR (replica.lag.time.max.ms and lastCaughtUpTimeMs), the AlterPartition request that tells the controller about ISR changes, the condition an acks=all request waits for as a DelayedProduce in the purgatory, the places where min.insync.replicas rejects a request, and how a follower truncates a diverged log with leader epochs after the leader changes — and we check it all with examples that freeze and kill brokers in a three-broker cluster.
-
Digging into Kafka's Source 10: KRaft and Metadata — How the Controller Remembers the Cluster
The cluster's ledger — topics, partitions, leaders, the list of brokers — is stored as records in __cluster_metadata, an ordinary Kafka log, and the controllers keep that log together with Raft. We follow it through the source: the roles in KafkaRaftClient (Prospective, Candidate, Leader, Follower) with Vote and Fetch, a high watermark set by a majority of voters, a QuorumController that runs on a single event thread and applies records to memory before they commit (and rolls them back with SnapshotRegistry), the MetadataLoader that turns records into MetadataDelta and MetadataImage on the broker, and heartbeats and fencing — checked with kafka-dump-log, kafka-metadata-quorum and a three-controller election.
-
Digging into Kafka's Source 11: The Consumer and the Group Coordinator — Inside poll() and Rebalancing
KafkaConsumer is a thin shell that hands every call to either ClassicKafkaConsumer or AsyncKafkaConsumer, depending on group.protocol. With the default, classic, the thread that calls poll() runs the rebalance itself through JoinGroup and SyncGroup, and a leader consumer computes the assignment. With the consumer protocol (KIP-848), a background thread talks to the broker with a single ConsumerGroupHeartbeat request and the broker decides the assignment. We follow both, then the broker-side group coordinator and its CoordinatorRuntime, which writes group state to __consumer_offsets as records and rebuilds it by reading them back — checked with an example that adds two consumers one after the other, and with kafka-dump-log.sh.
-
Digging into Kafka's Source 12: Idempotence and Transactions — How Exactly-Once Is Built, and Wrapping Up the Series
We follow in the source what happens behind enable.idempotence=true and a transactional.id. Where the broker uses the producer id, epoch and sequence to spot duplicates from retries; how the transaction coordinator records state in __transaction_state and has commit/abort markers (control records) written to every partition; how a read_committed consumer filters with the last stable offset (LSO) and a list of aborted transactions; and how an old producer with the same transactional.id is fenced off by its epoch — checked with raw-socket, kafka-dump-log and kafka-transactions examples. As the last post, it also looks back over parts 1–12 to close the series.