#request-processing
Posts
-
Digging into Kafka's Source 3: The Wire Protocol — Requests and Responses Generated from JSON Schemas
The bytes a producer and a broker exchange over TCP always have one shape, [4-byte length][request header][body]. We follow in the source how 203 JSON schemas that define that shape become 391 Java classes at build time, how the ApiVersions exchange picks a version, and how flexible versions and tagged fields let old and new versions live together, then check the bytes ourselves with an example that sends requests over a raw socket using nothing but the generated classes.
-
Digging into Kafka's Source 4: The Broker's Network Layer — Acceptors, Processors and Request Handlers
We follow, in SocketServer.scala, how the three kinds of threads in a broker's jstack — socket-acceptor, network-thread and request-handler — share the work of a single request. One Acceptor per listener; a Processor reads the bytes, puts the request on the request queue and mutes the connection; a request handler branches on KafkaApis.handle()'s match; and the response goes back to the Processor that read the request. With jstack, a raw-socket client and the request log we check how mute/unmute keeps one connection's requests in order by handling them one at a time, and where num.network.threads, num.io.threads and queued.max.requests take effect, and we compare it all with Tomcat's NIO connector.
-
Digging into Kafka's Source 6: How the Broker Accepts Records — The Record Batch Format and the Produce Path
A record batch sent by the producer is a 61-byte header followed by records squeezed with varints. We read those bytes straight out of a segment file, then follow one Produce request through KafkaApis → ReplicaManager → Partition → UnifiedLog → LogValidator → LogSegment with the source and the broker's TRACE log. Using hand-built Produce requests we confirm that the broker writes the batch almost as-is, fixing only a few header fields (the CRC stays the same), when it recompresses because the topic's compression.type differs, and where acks=0/1/all change the moment the response goes out.
-
Digging into Kafka's Source 8: The Read Path — Fetch Requests, Long Polling and Zero-Copy
We follow a consumer's Fetch request through the Kafka 4.5.0-SNAPSHOT broker — KafkaApis.handleFetchRequest() → ReplicaManager.fetchMessages() → UnifiedLog.read() → LogSegment.read() — until it becomes a slice of a file (FileRecords), how it waits in the purgatory (DelayedFetch) for up to fetch.max.wait.ms when there is not enough data, and how FileChannel.transferTo() finally sends it to the socket without passing through the JVM. Request logs, JMX and JFR examples show incremental fetch sessions, why consumers and followers share one Fetch, and why zero-copy disappears under TLS.
-
Digging into MariaDB's Source 5: The Protocol and Command Dispatch — How One Packet Reaches dispatch_command
How the bytes a client sends become packets, how a single first byte decides between COM_QUERY, COM_PING and COM_STMT_EXECUTE, and how the result turns back into packets on the way out — followed through sql/net_serv.cc, do_command() and dispatch_command() in sql_parse.cc, and protocol.cc. We talk to the server directly over bash's /dev/tcp with no client library, and run the query cache to see why it can look up a result before the SQL is even parsed.
-
Digging into MariaDB's Source 6: The Parser — How an SQL Statement Becomes LEX and Item Trees
We follow one SQL statement from a plain string through the lexer (Lex_input_stream), where it becomes tokens, into the rule actions of the Bison-generated parsers (MYSQLparse and ORAparse), where it becomes LEX, SELECT_LEX, TABLE_LIST and Item trees. We look at the build rule that turns one grammar file into two parsers, the keyword table and state_map, and the place the "near '...' at line N" error comes from, then print the trees the parser leaves behind with a Bison trace and gdb.
-
Digging into MariaDB's Source 7: Preparation — Checking Privileges, Opening Tables, Resolving Names
We follow, in the source, the three things the server does with the parser's LEX before it plans anything. In mysql_execute_command()'s big switch, check_table_access() checks privileges; open_and_lock_tables() takes metadata locks and pulls TABLE_SHARE and TABLE out of their caches; JOIN::prepare() binds column names to Field objects. We watch column-privilege errors 1142 and 1143, Opened_tables and Table_open_cache_hits moving, an ALTER TABLE waiting behind an open transaction with a plain SELECT queued behind it, and the call stack that raises errors 1054 and 1052.
-
Digging into MariaDB's Source 9: Execution and the Handler API — How the Nested-Loop Join Calls the Engine
We follow, in the source, the place where the optimizer's plan actually reads rows. From JOIN::exec() into the nested loop of do_select(), sub_select() and evaluate_join_record(), through the read functions chosen for each EXPLAIN type and the join buffer, past the handler's ha_ wrappers, and down to ha_innobase::rnd_next() and write_row(). We count calls with the Handler_read_* status variables, confirm the whole path from do_command to the engine with one gdb backtrace, and see that switching the engine to MEMORY, Aria or BLACKHOLE still goes through the same spots.
-
Digging into nginx's Source 8: Reading an HTTP Request — From a Connection to ngx_http_request_t
We follow, through the source, how one accepted socket becomes an ngx_http_request_t inside nginx — why the request struct is not created until data arrives, the switch-based state machines that parse the request line and headers, where client_header_buffer_size and large_client_header_buffers are used, the order in which the Host header picks a server block, and keep-alive and pipelining, all checked with debug logs, gdb and nc.
-
Digging into nginx's Source 9: The 11 Phases and Finding the location — In What Order a Request Is Handled
Once its headers are read, a request walks through 11 phases inside nginx. We check in the source how the handlers that modules register are flattened at config time into a single ph[] array, and how ngx_http_core_run_phases() drives that array through checker functions. We also see how location blocks become a prefix tree and are chosen in the order =, ^~, regex and 1.31.x's predicate locations, and which slot of the array rewrite last/break, internal redirects and named locations go back to — with debug logs and gdb.
-
Digging into Apache httpd's Source 10: Request Processing — From a URL to a File and a Handler
How does the single line GET /hello.html become a file on disk and a handler that sends it out? We follow the order of the phases in ap_process_request_internal() through the real source. We also look at the three walks that merge Location, Directory and Files settings, mod_dir's subrequest and ErrorDocument's internal redirect, using a module that logs every phase.
-
Digging into Tomcat's Source 6: Reading an HTTP/1.1 Request — From Bytes to a coyote Request
We follow, through the real source, how bytes read from a socket are split into a request line and headers inside the service() loop of Http11Processor and become a coyote Request. By sending raw requests we also see MessageBytes, which delays making Strings, the maxHttpHeaderSize and maxHeaderCount limits, how a bad request turns into an HTML 400 page, and where keep-alive and chunked responses are decided.
-
Digging into Tomcat's Source 7: CoyoteAdapter and the Mapper — Which Servlet Gets the Request
We follow CoyoteAdapter, the bridge that carries a fully read HTTP request from Coyote into Catalina; postParseRequest(), which decodes and normalizes the URI on the way; and the order of rules the Mapper uses to pick a Host, a Context and a Wrapper. Six servlets show the mapping priorities, curl shows what happens to %2F, .. and ;jsessionid, and a request object held too long shows why RequestFacade exists.
-
Digging into Tomcat's Source 8: Pipelines and Valves — The Processing Chain on Every Container
What does one <Valve> line in server.xml become in the source? We look at the pipeline each of Engine, Host, Context and Wrapper owns, the four basic valves at the ends that pick the next container, how a valve stops a request by not calling the next one, ErrorReportValve that draws the error page, the access log that is only written after the response has gone out, and the RemoteIpValve trap — checking it all against the source with our own valves hung on three containers.
-
Digging into Tomcat's Source 9: The Filter Chain and the Servlet Call — Until service() Runs
At the very end of the valve chain, Tomcat takes out a servlet instance, picks the filters that match this request, puts them in an array, and walks that array one step at a time until it finally calls service(). We follow this in the source, and use examples to check when a servlet's init() runs, how filter order is decided, how forward differs from include, and how an exception becomes an error page.