#record-batch
Posts
-
Digging into Kafka's Source 5: Inside the Producer — From send() to Batches and the Sender Thread
KafkaProducer.send() sends nothing over the network. It serializes the record, picks a partition, appends it to a per-partition batch in the RecordAccumulator and hands back a Future; the one thread that actually sends is kafka-producer-network-thread. We follow in the source the if in ready() where linger.ms and batch.size apply, the built-in partitioner that sticks keyless records to one partition, the BufferPool that hands out batch memory and max.block.ms, retries and delivery.timeout.ms, and max.in.flight.requests.per.connection, then check them with examples that count batches with kafka-dump-log.sh and print which thread runs the callback.
-
Digging into Kafka's Source 6: How the Broker Accepts Records — The Record Batch Format and the Produce Path
A record batch sent by the producer is a 61-byte header followed by records squeezed with varints. We read those bytes straight out of a segment file, then follow one Produce request through KafkaApis → ReplicaManager → Partition → UnifiedLog → LogValidator → LogSegment with the source and the broker's TRACE log. Using hand-built Produce requests we confirm that the broker writes the batch almost as-is, fixing only a few header fields (the CRC stays the same), when it recompresses because the topic's compression.type differs, and where acks=0/1/all change the moment the response goes out.