#consumer

Posts

• Digging into Kafka's Source 8: The Read Path — Fetch Requests, Long Polling and Zero-Copy

  Kafka · 2026-11-07

  We follow a consumer's Fetch request through the Kafka 4.5.0-SNAPSHOT broker — KafkaApis.handleFetchRequest() → ReplicaManager.fetchMessages() → UnifiedLog.read() → LogSegment.read() — until it becomes a slice of a file (FileRecords), how it waits in the purgatory (DelayedFetch) for up to fetch.max.wait.ms when there is not enough data, and how FileChannel.transferTo() finally sends it to the socket without passing through the JVM. Request logs, JMX and JFR examples show incremental fetch sessions, why consumers and followers share one Fetch, and why zero-copy disappears under TLS.

  #source-reading #zero-copy #consumer #request-processing

• Digging into Kafka's Source 11: The Consumer and the Group Coordinator — Inside poll() and Rebalancing

  Kafka · 2026-11-07

  KafkaConsumer is a thin shell that hands every call to either ClassicKafkaConsumer or AsyncKafkaConsumer, depending on group.protocol. With the default, classic, the thread that calls poll() runs the rebalance itself through JoinGroup and SyncGroup, and a leader consumer computes the assignment. With the consumer protocol (KIP-848), a background thread talks to the broker with a single ConsumerGroupHeartbeat request and the broker decides the assignment. We follow both, then the broker-side group coordinator and its CoordinatorRuntime, which writes group state to __consumer_offsets as records and rebuilds it by reading them back — checked with an example that adds two consumers one after the other, and with kafka-dump-log.sh.

  #source-reading #consumer #consumer-group #rebalance