#log-storage
Posts
-
Digging into Kafka's Source 6: How the Broker Accepts Records — The Record Batch Format and the Produce Path
A record batch sent by the producer is a 61-byte header followed by records squeezed with varints. We read those bytes straight out of a segment file, then follow one Produce request through KafkaApis → ReplicaManager → Partition → UnifiedLog → LogValidator → LogSegment with the source and the broker's TRACE log. Using hand-built Produce requests we confirm that the broker writes the batch almost as-is, fixing only a few header fields (the CRC stays the same), when it recompresses because the topic's compression.type differs, and where acks=0/1/all change the moment the response goes out.
-
Digging into Kafka's Source 7: Log Storage — Segments, Indexes, Retention and Compaction
When does a file like 00000000000000003830.log appear in a partition directory, how is one offset turned into a byte position in that file, who deletes a segment once retention.ms has passed, and how does cleanup.policy=compact sweep away old values of the same key? We follow UnifiedLog, LocalLog, LogSegment, OffsetIndex and LogCleaner in the source, and check with examples the five roll conditions, the sparse index that writes one entry per 4 KB and lives in mmap, the two-step delete through a .deleted rename, the SkimpyOffsetMap built from MD5 hashes of keys, and the recovery point used when a broker comes back from a crash.
-
Digging into Kafka's Source 9: Replication — The ISR, the High Watermark and the Wait Behind acks=all
A follower broker's ReplicaFetcherThread sends Fetch requests to the leader and copies its log, and the leader learns each follower's position from the offset in those Fetch requests and raises the high watermark. We follow, in Partition.scala and AbstractFetcherThread.scala, the rule for dropping out of the ISR (replica.lag.time.max.ms and lastCaughtUpTimeMs), the AlterPartition request that tells the controller about ISR changes, the condition an acks=all request waits for as a DelayedProduce in the purgatory, the places where min.insync.replicas rejects a request, and how a follower truncates a diverged log with leader epochs after the leader changes — and we check it all with examples that freeze and kill brokers in a three-broker cluster.