Digging into nginx's Source 8: Reading an HTTP Request — From a Connection to ngx_http_request_t

What this post covers

This is the eighth post in the "Digging into nginx's Source" series. In Part 7 we saw a worker's event loop wake up in epoll_wait, accept a connection with accept4, and call the handler function attached to that connection's read event. This post moves to the other side of that call — the HTTP module. We follow how the bytes arriving on a socket are split into one line, GET /hello?x=1 HTTP/1.1, plus a few header lines, and turned into a struct called ngx_http_request_t.

On the map from Part 2, this post zooms into the first box of the "C. One HTTP request" row: Reading the request.

The green box is this post. Once the request has been read, it is handed to the phase engine of Part 9.

We want to answer four questions.

• Right after accepting a connection, what does nginx create — and what does it deliberately not create? Why does the request struct appear only after the first byte arrives?
• How are the request line and headers split up, and how does the parser carry on when only half the data has arrived?
• Where are client_header_buffer_size and large_client_header_buffers used, and why does going over them produce 414 or "400 Request Header Or Cookie Too Large"?
• With several server {} blocks on one port, how does the Host header pick one — and how is the next request read on a keep-alive connection?

All line numbers in this post refer to this commit. How to fetch the same commit and build the Docker image used for the examples is in Part 1.

ngx_http_request.c is over 4,200 lines and ngx_http_parse.c over 2,400. This post picks only the main path of reading one HTTP/1.x request. Things that live in the same files but were left out — TLS, HTTP/2, the request body — are listed by name and location in "What this post leaves out" at the end.

Terms to know first

An event handler is "the function to call when this connection has something to read". nginx's connection struct ngx_connection_t has a read event c->read and a write event c->write, and each event's handler field is a function pointer. When a socket becomes readable, the event loop just calls rev->handler(rev) without knowing which function that is (Part 7). The plot of this post is the HTTP module swapping that function pointer at every stage.

ngx_http_connection_t and ngx_http_request_t are two structs with different lifetimes. The first is one per TCP connection and lives as long as the connection (which address:port it came in on, the list of large header buffers). The second is one per HTTP request; on a keep-alive connection it is created and destroyed for every request.

A memory pool (ngx_pool_t) is nginx's allocator that collects small allocations and frees them all at once. There is a connection pool per connection and a request pool per request, so when a request ends only its pool is thrown away. Allocations bigger than what a pool can cut off in one go are malloced separately and hung on a "large" list — that plays an important role in this post. The pool's structure was covered in Part 3.

A state machine is a code shape that keeps "which step am I at" in one variable (state) and, for every character read, decides the next step with switch (state). All of nginx's HTTP parsers look like this, and because state is stored in a struct, they can resume later from the exact point where the data ran out.

ngx_str_t is nginx's string type, with two fields: len and data. It doesn't have to end in NUL ('\0'), so it can point into the middle of a buffer without copying (Part 3).

Keep-alive and pipelining. Keep-alive means keeping the connection open after sending a response and waiting for the next request; pipelining means the client sends several requests at once without waiting for the responses. Both are "several requests on one connection", and in nginx the same function (ngx_http_set_keepalive()) handles both, in section 7.

1. Where a connection meets HTTP — ngx_http_init_connection()

The single line listen 8080; in nginx.conf becomes a listening socket when the config is read, and ngx_http_init_connection() is registered on that socket as "the function to call when a connection comes in".

    ls->handler = ngx_http_init_connection;

src/http/ngx_http.c line 1882

ngx_event_accept() from Part 7 accepts the new socket with accept4(), prepares the connection struct and the connection pool, and then calls this function (the ls->handler(c); at src/event/ngx_event_accept.c line 312). The event core knows nothing about HTTP, and the HTTP module knows nothing about accept; they are joined by this one function pointer. For a listen inside a stream {} block, ngx_stream_init_connection() goes in the same slot (Part 12).

ngx_http_init_connection() does three things.

1. It takes an ngx_http_connection_t from the connection pool and hangs it on c->data (lines 226–232).
2. It looks at the address:port the connection came in on, picks hc->addr_conf, and remembers that address's default server (default_server) configuration in hc->conf_ctx (lines 236–309). We haven't seen the Host header yet, so the default server it is for now. The real server block is chosen in section 6.
3. It sets the read handler and waits. Here is the end of the function.

    rev = c->read;
    rev->handler = ngx_http_wait_request_handler;
    c->write->handler = ngx_http_empty_handler;

    /* ... HTTP/3 takes another path; with ssl, rev->handler = ngx_http_ssl_handshake ... */

    if (rev->ready) {
        /* the deferred accept() */
        /* ... */
        rev->handler(rev);
        return;
    }

    cscf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_core_module);

    ngx_add_timer(rev, cscf->client_header_timeout);
    ngx_reusable_connection(c, 1);

    if (ngx_handle_read_event(rev, 0) != NGX_OK) {
        ngx_http_close_connection(c);
        return;
    }
}

src/http/ngx_http_request.c lines 328–373

• The read handler is ngx_http_wait_request_handler. With listen 443 ssl, ngx_http_ssl_handshake comes in first and hands over to the same waiting function once the handshake is done (a connection that picked h2 via ALPN goes to HTTP/2's ngx_http_v2_init() instead). We treat TLS only as "one step that slots in before the request".
• It sets a client_header_timeout timer (60 seconds by default). A client that connects and sends nothing is dropped after 60 seconds. The default comes from ngx_conf_merge_msec_value(conf->client_header_timeout, prev->client_header_timeout, 60000) in ngx_http_core_merge_srv_conf() (src/http/ngx_http_core_module.c lines 3626–3627).
• ngx_reusable_connection(c, 1) puts the connection on the list of "connections that aren't doing anything yet and may be closed in an emergency". When worker_connections runs out, nginx closes connections from this list first to make room (from src/core/ngx_connection.c line 1374, Part 7).

Here are all the read handlers this connection will go through, on one page. It is the map for the whole post.

Each box is the function sitting in c->read->handler at that moment. Sections 1–7 follow these boxes from top to bottom.

Easy to get wrong: what c->data holds keeps changing

data in ngx_connection_t is a void *, so it can hold anything. The HTTP module first stores an ngx_http_connection_t there (line 232, c->data = hc;), then an ngx_http_request_t once a request exists (line 527, c->data = ngx_http_create_request(c);), and an ngx_http_connection_t again when a keep-alive request ends (line 3424, c->data = hc;). That is why ngx_http_wait_request_handler() reads it as hc = c->data; (line 405) and ngx_http_process_request_line() as r = c->data; (line 1125). The same slot is interpreted as a different type depending on which handler is reading it, and the moment the handler changes always pairs with the moment c->data changes. When the request side needs the connection struct, it goes through r->http_connection.

2. No request until data arrives — ngx_http_wait_request_handler()

When the first byte arrives, the event loop calls ngx_http_wait_request_handler(). Only now is a buffer for the headers created.

    hc = c->data;
    cscf = ngx_http_get_module_srv_conf(hc->conf_ctx, ngx_http_core_module);

    size = cscf->client_header_buffer_size;

    b = c->buffer;

    if (b == NULL) {
        b = ngx_create_temp_buf(c->pool, size);
        /* ... */
        c->buffer = b;

    } else if (b->start == NULL) {
        /* ... */
    }

    size = b->end - b->last;

    n = c->recv(c, b->last, size);

    if (n == NGX_AGAIN) {
        /* ... re-arm the timer and the read event ... */
        if (b->pos == b->last) {

            /*
             * We are trying to not hold c->buffer's memory for an
             * idle connection.
             */

            if (ngx_pfree(c->pool, b->start) == NGX_OK) {
                b->start = NULL;
            }
        }

        return;
    }

src/http/ngx_http_request.c lines 405–463

The buffer size is client_header_buffer_size, 1024 bytes by default (ngx_http_core_module.c lines 3628–3629). c->recv() is a function pointer that calls recv() on the non-blocking socket; if there is nothing to read, it returns NGX_AGAIN instead of waiting.

The comment in the NGX_AGAIN branch states the design of this function: "we are trying to not hold c->buffer's memory for an idle connection." The 1024-byte buffer is bigger than what the connection pool (default connection_pool_size is 64 * sizeof(void *), 512 bytes on a 64-bit machine, lines 3622–3623) can cut off in one go, so ngx_palloc() mallocs it separately and puts it on the large list (src/core/ngx_palloc.c lines 123–132). And ngx_pfree() can give back only large allocations (lines 278–294). Because the buffer is a large allocation rather than part of a normal pool block, it can be returned without closing the connection.

Once a read succeeds, the request struct is created, the read handler is switched to the request-line function, and that function is called right away.

    c->log->action = "reading client request line";

    ngx_reusable_connection(c, 0);

    c->data = ngx_http_create_request(c);
    if (c->data == NULL) {
        ngx_http_close_connection(c);
        return;
    }

    rev->handler = ngx_http_process_request_line;
    ngx_http_process_request_line(rev);
}

src/http/ngx_http_request.c lines 523–535

Between the moment of connecting and the moment the first byte arrives, there is just one small struct.

Why put things off like this? A single nginx worker holds thousands or tens of thousands of connections at once. Many of them have connected but haven't sent anything yet, or are waiting on keep-alive for the next request. Give each of those a 1 KB buffer and a 4 KB request pool, and 10,000 connections tie up 50 MB for nothing. So the principle of this function is "only give memory to connections that have work to do". The same principle comes back with keep-alive in section 7.

What a new request struct starts with

ngx_http_create_request() builds the struct with ngx_http_alloc_request() and then counts this connection's requests with c->requests++ (line 550). Here is the core of the allocation.

    pool = ngx_create_pool(cscf->request_pool_size, c->log);
    /* ... */
    r = ngx_pcalloc(pool, sizeof(ngx_http_request_t));
    /* ... */
    r->pool = pool;

    r->http_connection = hc;
    r->signature = NGX_HTTP_MODULE;
    r->connection = c;

    r->main_conf = hc->conf_ctx->main_conf;
    r->srv_conf = hc->conf_ctx->srv_conf;
    r->loc_conf = hc->conf_ctx->loc_conf;

    r->read_event_handler = ngx_http_block_reading;

    r->header_in = hc->busy ? hc->busy->buf : c->buffer;
    /* ... create the headers_out list, the per-module ctx array, the variable values array ... */
    r->main = r;
    r->count = 1;

src/http/ngx_http_request.c lines 584–647

• The request pool is request_pool_size, 4096 bytes by default (lines 3624–3625). The struct itself lives inside this pool, so destroying the pool at the end of the request takes the struct with it.
• main_conf/srv_conf/loc_conf are three "arrays of config struct pointers, indexed by module number" (Part 5). For now they are copied from the default server picked in section 1. Macros like ngx_http_get_module_srv_conf(r, ngx_http_core_module) read these arrays, so swapping the array pointers changes every setting the request sees. Section 6 does exactly that.
• header_in is the buffer the request line and headers are read into. Normally it is the c->buffer from above, used as is — no copy.
• signature is a marker for debugging; its value NGX_HTTP_MODULE is the four letters "HTTP" read as an integer (the same trick as the module type values in Part 2).

ngx_http_request_s (ngx_http_request.h lines 386–614) is a big struct — its definition alone is nearly 230 lines. At the front come connection, the per-module ctx, the three config arrays above, and read_event_handler/write_event_handler (lines 389–397). These two are request-level handlers, separate from the connection's rev->handler. Once the request has been read, both of the connection's handlers are pinned to ngx_http_request_handler, which calls the request's two handlers on its behalf depending on the event type (end of section 6). After them come pool, header_in, headers_in (lines 407–411) and the results of the request line — method, http_version, request_line, uri, args, exten (lines 419–425) — and at the end of the struct sit the fields the parser uses: state, header_hash, and pointers such as uri_start, uri_end, args_start (lines 584–613). We use them in section 3.

Example 1: connect, then send two seconds later

We checked with the debug log that a request really is created only after data arrives. This example was run with nginx 1.31.9 built straight from the reference commit (9a7439fd79), in a Docker container (Debian bookworm, --with-debug). The config is a single location / { return 200 "uri=$uri args=$args\n"; } with error_log ... debug;, and the core of ex/u08-read-request/run.sh is: connect with nc, wait two seconds, then send the request.

( sleep 2; printf 'GET /hello?x=1 HTTP/1.1\r\nHost: a.test\r\nUser-Agent: nc\r\nConnection: close\r\n\r\n' ) | nc -q1 127.0.0.1 8080

Only the relevant lines of error.log are shown (dates removed).

02:54:26 [debug] 11#11: posix_memalign: 0000558749165BC0:512 @16
02:54:26 [debug] 11#11: *2 accept: 127.0.0.1:37958 fd:3
02:54:26 [debug] 11#11: *2 event timer add: 3: 60000:2346000
02:54:26 [debug] 11#11: *2 reusable connection: 1
02:54:28 [debug] 11#11: *2 http wait request handler
02:54:28 [debug] 11#11: *2 malloc: 0000558749168410:1024
02:54:28 [debug] 11#11: *2 recv: eof:0, avail:-1
02:54:28 [debug] 11#11: *2 recv: fd:3 76 of 1024
02:54:28 [debug] 11#11: *2 reusable connection: 0
02:54:28 [debug] 11#11: *2 posix_memalign: 0000558749162700:4096 @16
02:54:28 [debug] 11#11: *2 http process request line
02:54:28 [debug] 11#11: *2 http request line: "GET /hello?x=1 HTTP/1.1"
02:54:28 [debug] 11#11: *2 http uri: "/hello"
02:54:28 [debug] 11#11: *2 http args: "x=1"
02:54:28 [debug] 11#11: *2 http exten: ""
02:54:28 [debug] 11#11: *2 posix_memalign: 000055874918F320:4096 @16
02:54:28 [debug] 11#11: *2 http process request header line
02:54:28 [debug] 11#11: *2 http header: "Host: a.test"
02:54:28 [debug] 11#11: *2 http header: "User-Agent: nc"
02:54:28 [debug] 11#11: *2 http header: "Connection: close"
02:54:28 [debug] 11#11: *2 http header done
02:54:28 [debug] 11#11: *2 http finalize request: 0, "/hello?x=1" a:1, c:1
02:54:28 [debug] 11#11: *2 free: 0000558749162700, unused: 8
02:54:28 [debug] 11#11: *2 free: 000055874918F320, unused: 2640
02:54:28 [debug] 11#11: *2 reusable connection: 0
02:54:28 [debug] 11#11: *2 free: 0000558749168410
02:54:28 [debug] 11#11: *2 free: 0000558749165BC0, unused: 128

Times, addresses and the connection number (*2) differ on every run. The recv: eof: value varies too: on a re-run it printed eof:1, and after http finalize request a few lingering-close lines appeared (reusable connection: 1, event timer add: 3: 5000:..., recv: fd:3 0 of 4096). Whether the connection goes into the lingering close of section 7 seems to depend on when the client's FIN arrives. Here is what to read from it.

• 02:54:26 — right after connecting: the 512-byte connection pool (posix_memalign ...:512), accept, a 60000 ms timer, the reusable mark. That's all. No 1024-byte buffer and no 4096-byte request pool yet.
• 02:54:28 — data arrives two seconds later: only now does http wait request handler (line 392) appear; the 1024-byte buffer is malloced (it shows up as malloc because it is a large allocation), 76 bytes are read, and then the 4096-byte request pool is created.
• The request pool's second 4096-byte block was created where the request line is done and the header list is being prepared (line 1219, ngx_list_init(&r->headers_in.headers, ...)). The struct, the output header list and so on had nearly used up the first block, so the pool chained one more block (the block chain from Part 3).
• The end: the request pool's two blocks are freed (unused: 8, unused: 2640), and closing the connection frees the 1024-byte buffer (free: ...8410) and the connection pool.

The source lines that print these are: http process request line lines 1127–1128, http request line lines 1159–1160, http uri/args/exten lines 1387–1394, http header lines 1548–1550.

Example 2: call times and the stack in gdb

We also watched the same thing in gdb (ex/u08-gdb-stack). To make gdb easy, nginx ran as a single process with master_process off; daemon off;. We set breakpoints on three functions, printed the time at each hit, and let it continue. The client connects, waits two seconds, then sends the request.

[time] ngx_http_init_connection   03:24:59.514
[time] ngx_http_create_request    03:25:01.394
[time] ngx_http_parse_request_line 03:25:01.399
#0  ngx_http_parse_request_line at src/http/ngx_http_parse.c:141
#1  0x00005555555fadf4 in ngx_http_process_request_line at src/http/ngx_http_request.c:1149
#2  0x00005555555f9765 in ngx_http_wait_request_handler at src/http/ngx_http_request.c:534
#3  0x00005555555cea51 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:901
#4  0x00005555555bb0b9 in ngx_process_events_and_timers at src/event/ngx_event.c:248
#5  0x00005555555ca62b in ngx_single_process_cycle at src/os/unix/ngx_process_cycle.c:322
#6  0x000055555557f1b2 in main at src/core/nginx.c:400

(Argument lists were removed because they are long. Addresses and times differ on every run.) The gap between init_connection and create_request is about two seconds — the time the client waited. Reading the stack from the bottom up: main → the single-process cycle (with workers, ngx_worker_process_cycle sits here — Part 6) → the event loop from Part 7 → rev->handler(rev); at line 901 of epoll_module.c → the waiting handler → the request-line function called directly at line 534 → the parser. The first request line is read within the same call, without another trip around the event loop.

Where I got stuck. At first the client connected two seconds after nginx started, and the gap between the two times came out as 0.5 seconds instead of two. It looks like the connection landed in the kernel's queue (backlog) before nginx under gdb had finished starting, so accept happened late and the init_connection time was pushed back. Delaying the connection to five seconds gave the two-second gap above.

3. The request line — the ngx_http_parse_request_line() state machine

ngx_http_process_request_line() is a loop that repeats "read → parse". Here is the trimmed skeleton.

    rc = NGX_AGAIN;

    for ( ;; ) {

        if (rc == NGX_AGAIN) {
            n = ngx_http_read_request_header(r);

            if (n == NGX_AGAIN || n == NGX_ERROR) {
                break;
            }
        }

        rc = ngx_http_parse_request_line(r, r->header_in);

        if (rc == NGX_OK) {
            /* ... copy the request line into r's fields and clean up the URI ... */
            rev->handler = ngx_http_process_request_headers;
            ngx_http_process_request_headers(rev);

            break;
        }

        if (rc != NGX_AGAIN) {

            /* there was error while a request line parsing */

            ngx_log_error(NGX_LOG_INFO, c->log, 0,
                          ngx_http_client_errors[rc - NGX_HTTP_CLIENT_ERROR]);

            if (rc == NGX_HTTP_PARSE_INVALID_VERSION) {
                ngx_http_finalize_request(r, NGX_HTTP_VERSION_NOT_SUPPORTED);

            } else {
                ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST);
            }

            break;
        }

        /* NGX_AGAIN: a request line parsing is still incomplete */

        if (r->header_in->pos == r->header_in->end) {

            rv = ngx_http_alloc_large_header_buffer(r, 1);
            /* ... 500 on failure, 414 if it cannot grow any more ... */
        }
    }

src/http/ngx_http_request.c lines 1137–1273

The parser returns one of three things: NGX_OK (read up to the end of the line), NGX_AGAIN (looked at the whole buffer but the line isn't finished), or an error code (NGX_HTTP_PARSE_INVALID_METHOD and so on). The error codes are numbers starting at 10, so rc - NGX_HTTP_CLIENT_ERROR is directly an index into the ngx_http_client_errors[] array, from which a log sentence such as "client sent invalid method" is taken (lines 65–78).

On NGX_AGAIN the loop goes back to ngx_http_read_request_header() at the top, and that function doesn't read the socket blindly. If the buffer still has bytes the parser hasn't looked at, it uses those first, and it calls recv() only when the socket is marked readable (rev->ready) (lines 1610–1621). If there is nothing to read it returns NGX_AGAIN, the outer loop breaks too, and control goes back to the event loop. When more bytes arrive, the event loop calls rev->handler — this same ngx_http_process_request_line() — again. This time the parser does not start over; it resumes from the state where it stopped.

The parser's skeleton: state lives in the struct

    state = r->state;

    for (p = b->pos; p < b->last; p++) {
        ch = *p;

        switch (state) {

        /* HTTP methods: GET, HEAD, POST */
        case sw_start:
            r->request_start = p;

            if (ch == CR || ch == LF) {
                break;
            }

            if ((ch < 'A' || ch > 'Z') && ch != '_' && ch != '-') {
                return NGX_HTTP_PARSE_INVALID_METHOD;
            }

            state = sw_method;
            break;

        case sw_method:
            if (ch == ' ') {
                r->method_end = p - 1;
                m = r->request_start;
                state = sw_spaces_before_uri;

                switch (p - m) {

                case 3:
                    if (ngx_str3_cmp(m, 'G', 'E', 'T', ' ')) {
                        r->method = NGX_HTTP_GET;
                        break;
                    }
                    /* ... */

src/http/ngx_http_parse.c lines 141–175

At the start, state = r->state; fetches the saved step; when the buffer runs out, it is saved again at the end.

        }
    }

    b->pos = p;
    r->state = state;

    return NGX_AGAIN;

done:

    b->pos = p + 1;
    /* ... */
    r->http_version = r->http_major * 1000 + r->http_minor;
    r->state = sw_start;

src/http/ngx_http_parse.c lines 843–860

b->pos is also moved to "seen up to here", so the next call starts at the newly arrived bytes. When the line is complete, state goes back to sw_start. This one field, r->state, is shared with the header parser too, and returns to 0 every time a header line ends. In section 5 this "is it 0 or not" becomes important.

The enum has 27 states (sw_start through sw_almost_done, lines 111–139). Drawing only the path an ordinary GET /path HTTP/1.1 takes looks like this.

Each box is one case (boxes that group several are marked with "…"). The label on an arrow is the character that moves the parser to the next step.

A few techniques are worth noticing.

• Methods are split by length first. At the space, it switches on p - m (the method's length): 3 letters means only GET/PUT are compared, 4 letters POST/HEAD/COPY…. And the comparison isn't letter by letter either. On little-endian CPUs that allow unaligned reads, like x86, ngx_str3_cmp reads four bytes as one uint32_t and compares them in one go.

  #define ngx_str3_cmp(m, c0, c1, c2, c3)                                       \
      *(uint32_t *) m == ((c3 << 24) | (c2 << 16) | (c1 << 8) | c0)
  

  src/http/ngx_http_parse.c lines 42–43 (inside the #if (NGX_HAVE_LITTLE_ENDIAN && NGX_HAVE_NONALIGNED) at line 40; otherwise the letter-by-letter versions below line 72 are used)

• URI characters are filtered with a bitmap. usual[] is a table of eight 32-bit integers, one bit for each of the 256 characters (lines 17–37). If usual[ch >> 5] & (1U << (ch & 0x1f)) is true, the character is treated as "nothing special": in sw_check_uri the state stays the same (line 585; in sw_after_slash_in_uri, right after a /, line 525 moves on to sw_check_uri). Only characters like /, ., %, ?, # fall through to the switch for special handling.

• Complex URIs are only flagged. On seeing ., / or % right after a /, it sets the r->complex_uri or r->quoted_uri bit (lines 544–555) and moves on. The cleanup happens separately after the whole line has been read (below).

• Only HTTP major version 1 is accepted. A request line that says HTTP/2.0 becomes NGX_HTTP_PARSE_INVALID_VERSION immediately at http_major > 1 (lines 757–758) and ends with 505. HTTP/2 doesn't start with a request line: plain-text HTTP/2 is recognized by the preface at the very start of the connection and handed off separately in the function from section 2 (lines 500–521).

Example 3: sending the request line in two pieces

To see the saved state in action, we cut the request line between HT and TP/1.1 and sent the two pieces one second apart (the second part of ex/u08-read-request).

( printf 'GET /a/./b/../c.html?q=%%41 HT'; sleep 1; printf 'TP/1.1\r\nHost: a.test\r\nConnection: close\r\n\r\n' ) | nc -q1 127.0.0.1 8080

02:54:29 [debug] 11#11: *3 recv: eof:0, avail:-1
02:54:29 [debug] 11#11: *3 recv: fd:3 29 of 1024
02:54:29 [debug] 11#11: *3 http process request line
02:54:30 [debug] 11#11: *3 http process request line
02:54:30 [debug] 11#11: *3 recv: eof:0, avail:-1
02:54:30 [debug] 11#11: *3 recv: fd:3 43 of 995
02:54:30 [debug] 11#11: *3 http request line: "GET /a/./b/../c.html?q=%41 HTTP/1.1"
02:54:30 [debug] 11#11: *3 http uri: "/a/c.html"
02:54:30 [debug] 11#11: *3 http args: "q=%41"
02:54:30 [debug] 11#11: *3 http exten: "html"

• After reading the first 29 bytes (GET /a/./b/../c.html?q=%41 HT), http process request line appears once and then nothing happens for a second. The parser wrote the sw_http_HT state into r->state and went back to the event loop.
• A second later the same function is called again (the second http process request line), reads 43 more bytes into the remaining 995 bytes of the buffer (1024 − 29), and finishes the line. In other words, it appended to the same buffer.
• r->uri is /a/c.html and r->args is q=%41 as is. ./ and b/../ are gone, and the %41 in the query string was not decoded.

Cleaning up the URI — copy only when needed

When the request line is complete, ngx_http_process_request_uri() fills in r->uri.

    if (r->complex_uri || r->quoted_uri || r->empty_path_in_uri) {

        if (r->empty_path_in_uri) {
            r->uri.len++;
        }

        r->uri.data = ngx_pnalloc(r->pool, r->uri.len);
        if (r->uri.data == NULL) {
            ngx_http_close_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR);
            return NGX_ERROR;
        }

        cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module);

        if (ngx_http_parse_complex_uri(r, cscf->merge_slashes) != NGX_OK) {
            r->uri.len = 0;

            ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                          "client sent invalid request");
            ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST);
            return NGX_ERROR;
        }

    } else {
        r->uri.data = r->uri_start;
    }

src/http/ngx_http_request.c lines 1290–1315

For an ordinary URI it is one line, r->uri.data = r->uri_start; — it simply points into the buffer. Only for a complex URI does it reserve the same length in the request pool, and a second state machine, ngx_http_parse_complex_uri(), folds /./ and /../, decodes %XX, and — if merge_slashes is on (the default, line 3646) — collapses // into one, writing the result into the copy. The original (r->unparsed_uri) stays in the buffer and is used when the proxy passes the URI on unchanged.

Only the yellow boxes are actual memory. The rest are pointers and ngx_str_ts recording positions inside that memory.

This "just point at it" design is the same idea as MessageBytes in Tomcat Part 6. Tomcat also kept only a position and a length in the byte buffer and made a String for the first time on toString(). In C, nginx doesn't even need to make a string object — ngx_str_t is already the final form.

Easy to get wrong: .. above the root is a 400

When .. climbs above the root, as in /a/../../etc/passwd, ngx_http_parse_complex_uri() walks back looking for the previous /, passes the start of the copy (r->uri.data), and returns NGX_HTTP_PARSE_INVALID_REQUEST on the spot (src/http/ngx_http_parse.c lines 1512–1519). So this request ends with 400 at the request-line stage, before any location is even looked up (item 3 in the section 5 example). The path to a file outside root is blocked first by the parser. The log sentence is the same "client sent invalid request" as for request-line parser errors, so you can't tell the two apart from the log alone.

4. Headers — line by line, with a hash to find the handler

When the request line is done, the read handler switches to ngx_http_process_request_headers(). It has the same "read → parse" loop shape, and the parsing is done by ngx_http_parse_header_line(), one header line at a time. Every time a line is finished (NGX_OK), this happens.

        /* the host header could change the server configuration context */
        cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module);

        rc = ngx_http_parse_header_line(r, r->header_in,
                                        cscf->underscores_in_headers);

        if (rc == NGX_OK) {

            r->request_length += r->header_in->pos - r->header_name_start;

            if (r->invalid_header && cscf->ignore_invalid_headers) {
                /* ... only log "client sent invalid header line" and skip it ... */
                continue;
            }
            /* ... max_headers check ... */
            h = ngx_list_push(&r->headers_in.headers);
            /* ... */
            h->hash = r->header_hash;

            h->key.len = r->header_name_end - r->header_name_start;
            h->key.data = r->header_name_start;
            h->key.data[h->key.len] = '\0';

            h->value.len = r->header_end - r->header_start;
            h->value.data = r->header_start;
            h->value.data[h->value.len] = '\0';
            /* ... prepare the lowercase name h->lowcase_key ... */
            hh = ngx_hash_find(&cmcf->headers_in_hash, h->hash,
                               h->lowcase_key, h->key.len);

            if (hh && hh->handler(r, h, hh->offset) != NGX_OK) {
                break;
            }

src/http/ngx_http_request.c lines 1480–1546

1. Every header goes into the list. An ngx_table_elt_t is appended to r->headers_in.headers (an ngx_list_t), and its key and value point into the buffer. Then the buffer byte right after the name and after the value (:, CR, a space, …) is overwritten with '\0'. By writing into the buffer instead of copying, the strings can be passed straight to C string functions.
2. The parser has already computed the hash while reading. For every character of the header name it lowercases the character and updates the hash (hash = ngx_hash(hash, c);, ngx_http_parse.c lines 963–971), so the answer is already in r->header_hash when the line ends. The name is not scanned again.
3. For a known header, its handler is called. headers_in_hash is a hash built from the ngx_http_headers_in[] table when the config is read (src/http/ngx_http.c lines 427–446). The table looks like this.

ngx_http_header_t  ngx_http_headers_in[] = {
    { ngx_string("Host"), offsetof(ngx_http_headers_in_t, host),
                 ngx_http_process_host },

    { ngx_string("Connection"), offsetof(ngx_http_headers_in_t, connection),
                 ngx_http_process_connection },

src/http/ngx_http_request.c lines 81–86

Each entry is { name, offset within ngx_http_headers_in_t, handler } (ngx_http_request.h lines 173–177). ngx_http_headers_in_t has an ngx_table_elt_t * field for each frequently used header, such as host, connection, content_length, user_agent (lines 186–260), so other modules get them directly as r->headers_in.host without searching the list.

Most handlers reuse the same few functions. The most common, ngx_http_process_header_line(), just finds the field by offset and hangs the header on it.

    ph = (ngx_table_elt_t **) ((char *) &r->headers_in + offset);

    while (*ph) { ph = &(*ph)->next; }

    *ph = h;
    h->next = NULL;

src/http/ngx_http_request.c lines 1830–1835

(char *) &r->headers_in + offset means "the field that many bytes from the start of the struct". One function, with only the offset changing, handles Referer, Content-Type, Cookie and more. If the same name comes again, it is chained via next. Headers that must not appear twice, such as Content-Length, Transfer-Encoding and Authorization, are handled by ngx_http_process_unique_header_line() instead, which logs "client sent duplicate header line" on the second one and ends with 400 (lines 1841–1863). Host has its own function, ngx_http_process_host(), which we see in section 6.

Every header goes into the list; only headers in the table go through a handler. The three examples are picked from the ngx_http_headers_in[] table (ngx_http_request.c from line 81).

The empty line — checks when the headers end

On the empty line (CR LF), the parser returns NGX_HTTP_PARSE_HEADER_DONE and ngx_http_process_request_header() looks at the headers as a whole. HTTP/1.1 without Host is a 400 with "client sent HTTP/1.1 request without "Host" header" (src/http/ngx_http_request.c lines 2035–2040). Then come checks up to line 2122: converting Content-Length to a number, 400 when Content-Length and Transfer-Encoding: chunked come together, 501 for an unknown Transfer-Encoding, 405 for the TRACE method. Ambiguous combinations that could be abused for request smuggling are cut off here.

Easy to get wrong: headers with underscores silently disappear

A header whose name contains _, like X_Custom_Id: 1, is marked r->invalid_header = 1 by the parser when underscores_in_headers is off (the default, lines 3648–3649) (ngx_http_parse.c lines 973–983). And when ignore_invalid_headers is on (the default, lines 3643–3644), the continue in the code above skips it without even putting it in the list. It's not an error and the response is normal, which makes it a hard cause to find when a backend says "the header never arrived". The only trace is one info-level line, "client sent invalid header line". The limit on the number of header lines, max_headers (default 1000, line 3641), is a directive that also exists in 1.30.6.

5. When the buffer runs out — client_header_buffer_size and large_client_header_buffers

What happens when the headers don't fit in the 1024-byte buffer? Start with the defaults of the two settings.

    ngx_conf_merge_size_value(conf->client_header_buffer_size,
                              prev->client_header_buffer_size, 1024);
    ngx_conf_merge_bufs_value(conf->large_client_header_buffers,
                              prev->large_client_header_buffers,
                              4, 8192);

src/http/ngx_http_core_module.c lines 3628–3632

It starts with one 1k buffer and, when that runs out, uses up to four more 8k "large buffers". ngx_http_alloc_large_header_buffer(), called from the loops in sections 3 and 4 when r->header_in->pos == r->header_in->end (the whole buffer has been looked at but the line isn't over), does that job.

    old = request_line ? r->request_start : r->header_name_start;

    cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module);

    if (r->state != 0
        && (size_t) (r->header_in->pos - old)
                                     >= cscf->large_client_header_buffers.size)
    {
        return NGX_DECLINED;
    }

    hc = r->http_connection;

    if (hc->free) {
        cl = hc->free;
        hc->free = cl->next;
        /* ... */
    } else if (hc->nbusy < cscf->large_client_header_buffers.num) {

        b = ngx_create_temp_buf(r->connection->pool,
                                cscf->large_client_header_buffers.size);
        /* ... */
    } else {
        return NGX_DECLINED;
    }

    cl->next = hc->busy;
    hc->busy = cl;
    hc->nbusy++;

src/http/ngx_http_request.c lines 1679–1727

8k and 4 are the defaults. The point is that there are two roads to the red box: one line being too long, and all four buffers used up.

• There are two limits: "one line" and "number of buffers". If one line (the request line or a single header line) exceeds one large buffer (8k), it fails. Even with short lines, it also fails once the headers fill all four buffers. So the "total header limit" is roughly 1k + 8k × 4, but not exactly, because when a line straddles buffers the leftover space is abandoned and the line moved.
• Only the unfinished line is moved. With a new buffer, it memcpys only from old (the start of the line being read) to pos to the front of the new buffer, and recomputes every pointer such as request_start, uri_start, header_name_start for the new address (lines 1750–1816). The earlier, finished lines stay in the old buffer, and the ngx_table_elt_ts keep pointing there. That's why the old buffer is not freed.
• If r->state == 0, nothing is copied. If the buffer filled up exactly where a line had just ended, there is nothing to move, so it just switches to the new buffer (lines 1729–1739). This is where "is state 0" from section 3 is used. For the request line with state 0, the client has filled the buffer with nothing but empty lines (CR LF), so it simply rewinds the buffer to the start (lines 1669–1677).
• Large buffers belong to the connection. hc->busy/hc->free live in ngx_http_connection_t because of pipelining: the next request may already be sitting at the end of a large buffer, so the buffers can't be thrown away as soon as a request ends (section 7).

On failure, the two functions pick different status codes. The request-line side uses NGX_HTTP_REQUEST_URI_TOO_LARGE (414, lines 1263–1270), the header side NGX_HTTP_REQUEST_HEADER_TOO_LARGE (lines 1444–1469). The latter is 494, an nginx-specific code not in the HTTP standard (ngx_http_request.h lines 117–119). 494 is not sent to the client as is: when ngx_http_special_response_handler(), which builds error pages, sees 494 (or other nginx-specific codes such as 497), it changes the status to 400 with r->err_status = NGX_HTTP_BAD_REQUEST; (src/http/ngx_http_special_response.c lines 513–524). The body uses the dedicated 494 page (ngx_http_error_494_page, lines 252–258), so the status is 400 but the title says "Request Header Or Cookie Too Large". Having a separate code means error_page 494 ...; can show a different page for just this case.

Example 4: bad requests and big headers

We sent raw requests with nc and printed the first line of the response plus the [info] lines from error.log (ex/u08-bad-requests, buffer settings left at their defaults). Each request is sent like this.

printf 'get / HTTP/1.1\r\nHost: a\r\n\r\n' | nc -q1 127.0.0.1 8080 | head -1

Dates and times, and everything after , client: ..., were removed from the log lines, and long lines were cut at 150 characters.

=== 1. lowercase method ===
HTTP/1.1 400 Bad Request
[info] 11#11: *2 client sent invalid method while reading client request line
=== 2. HTTP/2.0 in request line ===
HTTP/1.1 505 HTTP Version Not Supported
[info] 11#11: *3 client sent invalid version while reading client request line
=== 3. URI goes above root ===
HTTP/1.1 400 Bad Request
[info] 11#11: *4 client sent invalid request while reading client request line
=== 4. HTTP/1.1 without Host ===
HTTP/1.1 400 Bad Request
[info] 11#11: *5 client sent HTTP/1.1 request without "Host" header while reading client request headers
=== 5. duplicate Host ===
HTTP/1.1 400 Bad Request
[info] 11#11: *6 client sent duplicate host header: "Host: b", previous value: "Host: a" while reading client request headers
=== 6. 9000-byte header line ===
HTTP/1.1 400 Bad Request
[info] 11#11: *7 client sent too long header line: "X-Long: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
=== 7. 9000-byte URI ===
HTTP/1.1 414 Request-URI Too Large
[info] 11#11: *8 client sent too long URI while reading client request line
=== 8. 2000-byte header line (fits in a large buffer) ===
HTTP/1.1 200 OK
[debug] 11#11: *9 recv: fd:3 1024 of 1024
[debug] 11#11: *9 http alloc large header buffer
[debug] 11#11: *9 http large header alloc: 000055A4B2BEF330 8192
[debug] 11#11: *9 http large header copy: 999
[debug] 11#11: *9 recv: fd:3 1031 of 7193
=== 9. 494 page body ===
<html>
<head><title>400 Request Header Or Cookie Too Large</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>Request Header Or Cookie Too Large</center>

• 1–3 failed at the request-line stage ("while reading client request line" at the end of the log line), 4–6 at the header stage. That tail is the string stored in c->log->action, swapped in at line 523 in section 2 and line 1227 in section 3, so the log records which stage failed for free.
• 2 is 505. The parser rejected major version 2 (section 3).
• 6 and 7 are both 9000 bytes, yet the header gets 400 (494 inside) and the URI gets 414. Item 9 is the body of the same request as item 6.
• 8 succeeds. The first recv filled the 1024-byte buffer and the X-Mid: line wasn't finished, so it allocated a new 8192-byte large buffer (large header alloc), moved the 999 unfinished bytes (large header copy: 999), and read on into the remaining 7193 bytes (8192 − 999). 999 is 1024 minus GET / HTTP/1.1\r\n (16 bytes) and Host: a\r\n (9 bytes) — proof that the two finished lines before it were not moved. The debug strings come from lines 1666–1667, lines 1717–1719 and lines 1741–1742.

Compared with httpd: httpd measures the request line (LimitRequestLine) and a single header line (LimitRequestFieldSize) separately, 8190 bytes each, and an oversized request line gives 414 (httpd Part 9). nginx is similar — 8k per line, 414 for the request line — but it has a second limit, the number of buffers. Tomcat measures the request line and headers against one limit (maxHttpHeaderSize) and returns 400 for both an oversized URI and oversized headers (Tomcat Part 6).

6. Which server block? — choosing the virtual server

With several server {} blocks on one port, requests arriving on the same address:port have to be told apart by the Host header. In section 1 the connection got the default server for now. The moment the Host header line is read, its handler ngx_http_process_host() picks the real server (lines 1866–1915). First ngx_http_validate_host() cleans up the value: if there are uppercase letters it makes a lowercase copy in the request pool, strips a port such as :8080 into r->port, and also drops a single trailing dot (example.com.) (lines 2276–2471). Then it calls ngx_http_set_virtual_server() — unless the request line was an absolute URI like GET http://host/ HTTP/1.1 that already picked a server (lines 1178–1203), in which case the Host header doesn't choose again (lines 1903–1905).

ngx_http_set_virtual_server() delegates the search to ngx_http_find_virtual_server() and, if a server is found, swaps two config array pointers.

    if (rc == NGX_DECLINED) {
        return NGX_OK;
    }

    r->srv_conf = cscf->ctx->srv_conf;
    r->loc_conf = cscf->ctx->loc_conf;

src/http/ngx_http_request.c lines 2542–2547

In the source, "choosing a server block" is these two lines. As we saw in section 2, every module reads its settings through these arrays, so from then on every value ngx_http_get_module_loc_conf(r, ...) returns for this request belongs to the chosen server block. If nothing is found (NGX_DECLINED), nothing changes and the default server stays.

The search order is set by ngx_hash_find_combined(), which ngx_http_find_virtual_server() calls: the exact-name hash, then the leading wildcard (*.example.com), then the trailing wildcard (www.example.*) (src/core/ngx_hash.c lines 211–245; the call site is ngx_http_request.c lines 2568–2575). If none match, the regexes are tried one by one in the order they appear in the config (lines 2620–2634).

The three yellow boxes are hash lookups, so they stay fast with many server blocks; only the grey regexes are tried one at a time.

The name hash is built by ngx_http_server_names() when the config is read, but only when that address:port has two or more server blocks (or when the default server has a regex server_name with captures) (src/http/ngx_http.c lines 1572–1581). With just one there is nothing to choose, so virtual_names is NULL (lines 1973–1984) and ngx_http_find_virtual_server() returns on its first line.

Example 5: which server block answers for which Host

We put five server blocks on one port, 8080 (ex/u08-keepalive-vhost).

server { listen 8080 default_server; server_name _;               return 200 "default    host=$host\n"; }
server { listen 8080;                server_name www.example.com; return 200 "exact      host=$host\n"; }
server { listen 8080;                server_name *.example.com;   return 200 "wc-head    host=$host\n"; }
server { listen 8080;                server_name www.example.*;   return 200 "wc-tail    host=$host\n"; }
server { listen 8080;                server_name ~^api(?<n>\d+)\.example\.org$;
                                                                  return 200 "regex n=$n host=$host\n"; }

(Condensed to one line each for the post; the real file spreads each block over several lines.) Here are the results of sending curl -H "Host: ...".

www.example.com        -> exact      host=www.example.com
WWW.Example.COM:8080   -> exact      host=www.example.com
a.example.com          -> wc-head    host=a.example.com
www.example.net        -> wc-tail    host=www.example.net
api42.example.org      -> regex n=42 host=api42.example.org
unknown.test           -> default    host=unknown.test
(HTTP/1.0, no Host)    -> default    host=_

• WWW.Example.COM:8080, with uppercase letters and a port, was still found by exact name — the result of validate_host lowercasing it and removing the port. $host is the cleaned-up value too.
• www.example.com also matches the leading wildcard *.example.com, but the exact name wins.
• The regex's named capture (?<n>...) became the variable $n. Storing captures as variables when a regex server is chosen is done by ngx_http_regex_exec().
• The HTTP/1.0 request without Host went to the default server, and $host is then the first server_name of the default server (_). As HTTP/1.1 it would have been a 400 from the check in section 4.

Finishing the read — ngx_http_process_request()

Once everything up to the empty line has been read and checked, ngx_http_process_request() is called. The TLS client certificate checks (ssl_verify_client) are at the top of this function; the end looks like this.

    if (c->read->timer_set) {
        ngx_del_timer(c->read);
    }

#if (NGX_STAT_STUB)
    (void) ngx_atomic_fetch_add(ngx_stat_reading, -1);
    /* ... */
    (void) ngx_atomic_fetch_add(ngx_stat_writing, 1);
    /* ... */
#endif

    c->read->handler = ngx_http_request_handler;
    c->write->handler = ngx_http_request_handler;
    r->read_event_handler = ngx_http_block_reading;

    if (ngx_http_read_early_body(r) != NGX_OK) {
        return;
    }

    ngx_http_handler(r);
}

src/http/ngx_http_request.c lines 2198–2218

• The client_header_timeout timer is deleted. That timer covers "until the headers are fully read". While the response is being produced, other timers (send_timeout, upstream timers) take over.
• stub_status's Reading goes down by one and Writing up by one. The "Reading: N" on a stub_status page was the number of requests currently going through sections 1–6 of this post.
• Both connection handlers are pinned to ngx_http_request_handler. This function is a relay that calls r->write_event_handler(r) for a write event and r->read_event_handler(r) for a read event (lines 2643–2677). From now on, changing stages means changing the request's pointers, not the connection's. On the read side, ngx_http_block_reading means "not interested in reading right now".
• ngx_http_read_early_body() is a new 1.31.x feature that reads the request body before entering the phases when the client_body_early_read directive is enabled (the directive does not exist in 1.30.6's ngx_http_core_module.c). It is off by default (NULL, lines 3651–3652), so it normally returns NGX_OK right away.
• ngx_http_handler(r) is the entrance to the phase engine. It sets r->write_event_handler = ngx_http_core_run_phases; and starts running the phases — that is the story of Part 9.

7. The next request — keep-alive and pipelining

Who decides keep-alive

Right after the request has been read, ngx_http_handler() sets the initial value of r->keepalive (src/http/ngx_http_core_module.c lines 856–870). If the Connection header handler from section 4 has set connection_type (by checking whether the value contains close or keep-alive, lines 1918–1934), it follows that; with no header it is r->http_version > NGX_HTTP_VERSION_10 — on for HTTP/1.1, off for 1.0. Later, once the location is known, it is switched off for keepalive_timeout 0, when the connection's request count reaches keepalive_requests (default 1000), or when the connection is older than keepalive_time (default one hour) (lines 1390–1400; defaults at lines 3986–3995). This is where the c->requests++ from section 2 is used. Some error responses, such as 400, 414 and 500, switch keep-alive off even when r->keepalive is set (ngx_http_special_response.c lines 436–447). Errors that happen before the request has been fully read, like those in the section 5 example, never reach ngx_http_handler(), so r->keepalive is 0 from the start. The source doesn't say why, but it is presumably because the bytes left after a broken request can't be trusted as the start of the next one.

When the response is finished, ngx_http_finalize_connection() looks at r->keepalive and decides whether to go to ngx_http_set_keepalive(), to a lingering close, or to just close (lines 2999–3075).

ngx_http_set_keepalive() — drop the request, keep the connection

The heart of this function is freeing the request struct (ngx_http_free_request() → the request pool is destroyed, line 3422) and putting hc back in c->data (line 3424). Then it splits two ways.

If the buffer already holds bytes of the next request (pipelining), it creates a new request right away without waiting.

    if (b->pos < b->last) {

        ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "pipelined request");

        c->log->action = "reading client pipelined request line";

        r = ngx_http_create_request(c);
        if (r == NULL) {
            ngx_http_close_connection(c);
            return;
        }

        r->pipeline = 1;

        c->data = r;
        /* ... */
        rev->handler = ngx_http_process_request_line;
        ngx_post_event(rev, &ngx_posted_events);
        return;
    }

src/http/ngx_http_request.c lines 3434–3461

ngx_post_event() puts the event on the posted queue, meaning "call this handler before the current turn of the event loop ends" (Part 7). Rather than calling it right away on the current call stack, it defers through the queue so the next request starts cleanly after the stack that was finishing the response has fully unwound.

If the buffer is empty, it gives memory back on the same principle as section 2. It ngx_pfree()s the 1k buffer, frees all the large header buffers (hc->free, hc->busy) as well (lines 3470–3513), switches the read handler to ngx_http_keepalive_handler and sets a keepalive_timeout timer (75 seconds by default) (lines 3521–3571). A connection waiting on keep-alive ends up holding only the 512-byte connection pool and the ngx_http_connection_t.

When bytes of the next request arrive, ngx_http_keepalive_handler() re-allocates the buffer (lines 3636–3653), reads, and then finishes exactly like the end of section 2: it creates a new request with c->data = ngx_http_create_request(c);, switches to rev->handler = ngx_http_process_request_line; and calls it at once (lines 3711–3724).

The request is dropped and the connection kept; then what's left in the buffer decides between two paths. The green boxes are the two connections in Example 6 below.

Example 6: two requests on one connection — keep-alive and pipelining

In the same example folder, we sent two requests over one connection twice. The first time, one request, a one-second pause, then the second request (with Connection: close); the second time, both requests back to back in a single printf.

=== 2. keep-alive: two requests, 1s apart, one connection ===
HTTP/1.1 200 OK
exact      host=www.example.com
HTTP/1.1 200 OK
wc-head    host=a.example.com
--- error.log (filtered) ---
03:24:43 [debug] 11#11: *9 accept: 127.0.0.1:37362 fd:3
03:24:43 [debug] 11#11: *9 http wait request handler
03:24:43 [debug] 11#11: *9 http request line: "GET /first HTTP/1.1"
03:24:43 [debug] 11#11: *9 set http keepalive handler
03:24:43 [debug] 11#11: *9 hc free: 0000000000000000
03:24:43 [debug] 11#11: *9 hc busy: 0000000000000000 0
03:24:44 [debug] 11#11: *9 http keepalive handler
03:24:44 [debug] 11#11: *9 http request line: "GET /second HTTP/1.1"
03:24:44 [debug] 11#11: *9 close http connection: 3
=== 3. pipelining: two requests in one write ===
HTTP/1.1 200 OK
exact      host=www.example.com
HTTP/1.1 200 OK
exact      host=www.example.com
--- error.log (filtered) ---
03:24:47 [debug] 11#11: *10 accept: 127.0.0.1:33142 fd:3
03:24:47 [debug] 11#11: *10 http wait request handler
03:24:47 [debug] 11#11: *10 http request line: "GET /p1 HTTP/1.1"
03:24:47 [debug] 11#11: *10 set http keepalive handler
03:24:47 [debug] 11#11: *10 pipelined request
03:24:47 [debug] 11#11: *10 http request line: "GET /p2 HTTP/1.1"
03:24:47 [debug] 11#11: *10 close http connection: 3

• The connection number *9 stays the same for both requests. One accept, two request lines: two request structs were created and destroyed one after the other on one connection.
• On the keep-alive side, after the first response come set http keepalive handler (line 3367) → empty hc free/hc busy since no large buffers were used (lines 3486–3501) → one second later the second request arrives and http keepalive handler (line 3591) creates a new request. The second request carried Host: a.example.com, so a different server block (wc-head) answered on the same connection. Server blocks are chosen per request, not per connection.
• On the pipelining side, pipelined request (line 3436) comes right after set http keepalive handler, and GET /p2 is read with neither http wait request handler nor http keepalive handler. The second request's bytes were already in the buffer, so there was no need to wait on the socket.

Lingering close, briefly

When it isn't keep-alive but the client may still be sending (an error response sent before the body was fully read, for instance), nginx doesn't close the socket right away. Calling close() with unread data pending makes the kernel send an RST, and the client may then throw away even the part of the response it hasn't received yet (this is general TCP behavior, not an explanation found in a comment in this version's source). So ngx_http_set_lingering_close() closes only the write side (shutdown(SHUT_WR), line 3780), reads and discards incoming input for lingering_timeout, and then closes (lines 3727–3795). Which cases go to a lingering close is decided by the lingering_close directive and lines 3063–3072 of ngx_http_finalize_connection() above.

What this post leaves out

• The TLS handshake and choosing a server by SNI (ngx_http_ssl_handshake, ngx_http_ssl_servername): from src/http/ngx_http_request.c line 675 and line 886
• Reading the PROXY protocol header: lines 479–498, src/core/ngx_proxy_protocol.c
• Spotting the HTTP/2 preface and handing over to ngx_http_v2_init(): lines 500–521, src/http/v2/. HTTP/3: src/http/v3/
• All the states of ngx_http_parse_complex_uri(): from src/http/ngx_http_parse.c line 1300
• The CONNECT method and allow_connect (a directive not present in 1.30.6's ngx_http_core_module.c): the parser's sw_spaces_before_host, ngx_http_request.c lines 2098–2112
• Reading the request body (ngx_http_read_client_request_body): src/http/ngx_http_request_body.c — briefly in Part 11
• keepalive_min_timeout (a directive that also exists in 1.30.x): lines 3556–3571
• Finishing and freeing a request (ngx_http_finalize_request, ngx_http_free_request): from line 2743 and line 3981

Summary

What's next

The request has been read. ngx_http_request_t now holds the method, URI, headers and the chosen server block's settings, and ngx_http_handler() has started the phase engine. The next post goes inside it: in what order a request passes through the 11 phases from NGX_HTTP_POST_READ_PHASE to NGX_HTTP_LOG_PHASE, by which rules a location is chosen along the way, and how rewrite ... last sends the request back a phase. It continues in Part 9: The 11 Phases and Finding the location — In What Order a Request Is Handled.

We've seen the same job in earlier series. In httpd Part 9, httpd read the request line and headers with filters, and while the event MPM releases the thread between requests, one worker thread looked after the connection while a request was being read and processed. nginx has no threads for this: it saves the parser's state in a struct and returns to the event loop, so even with a half-arrived request the worker keeps serving other connections. This "parser that remembers where it stopped" is the same solution as the phase number carried around by parseRequestLine() in Tomcat Part 6 — anyone who wants non-blocking reads ends up doing it this way.

• Part 7: The Event Loop — epoll, Timers, Posted Events and accept
• Part 3: Memory Pools and Core Data Structures — ngx_pool_t, ngx_str_t, ngx_buf_t and ngx_chain_t
• Part 5: Modules and Directives — How One Line of nginx.conf Becomes a Module's Config Struct
• Apache httpd Part 9: Connections and Reading Requests
• Tomcat Part 6: Reading HTTP/1.1 Requests — How Bytes Become a coyote Request