Digging into Apache httpd's Source 9: Connections and Reading a Request — From a Socket to a request_rec

What this post covers

This is the ninth post in the "Digging into Apache httpd's Source" series. Part 8 looked at how an MPM organises processes and threads, accepts connections and hands them to workers. This post starts right after that, at the moment a worker has a socket in its hand.

There are four questions to answer.

• What functions and hooks does an accepted socket pass through before it reaches "the code that reads an HTTP request"?
• Keep-alive lets one connection carry several requests. Where in the code is that repetition?
• Who reads the request line (GET /index.html HTTP/1.1) and the headers, and where are directives like LimitRequestLine and HttpProtocolOptions used?
• When several virtual hosts share one IP and port, when and how is one of them chosen?

Every line number and every detail in this post refers to that commit. Part 1 explains how to get the source at the same commit and set up the example environment. This post keeps only the main thread that someone opening the source for the first time can follow; what was left out is named at the end, under "What this post leaves out".

Words to know first

conn_rec and request_rec. httpd represents one TCP connection as one conn_rec struct, and one HTTP request arriving on that connection as one request_rec. A connection can carry several requests, so one conn_rec gets several request_recs, one after another. Part 3 showed both structs in full. This post looks at when, and by whom, the fields of a request_rec get filled in.

Pool. httpd does not free memory piece by piece. It takes memory from a "pool" and throws the whole pool away at once. A connection has a connection pool, and each request has a request pool that is a child of it, so when a request ends only its pool is dropped, and when the connection ends the connection pool goes as a whole. Part 4 has the details.

Hooks, RUN_FIRST and RUN_ALL. A hook is a point where modules register functions to be called "when this moment comes". A RUN_FIRST hook calls the registered functions in turn and stops as soon as one returns something other than DECLINED; a RUN_ALL hook keeps calling them as long as they return OK or DECLINED (any other value stops it). The order is set with values like APR_HOOK_FIRST, MIDDLE, LAST and REALLY_LAST. Part 6 unpacks the macros behind them.

Request line and headers. An HTTP/1.x request is lines of text. The first one is the request line (method, address, protocol version); below it come headers of the form name: value, ending with one empty line. Each line ends with the two characters \r\n (CR LF).

GET /index.txt?x=1 HTTP/1.1
Host: b.example
User-Agent: raw
Connection: close

Keep-alive. After sending a response, the server keeps the connection open and takes the next request on it. In HTTP/1.1 keep-alive is the default; in httpd it is tuned with the KeepAlive, KeepAliveTimeout and MaxKeepAliveRequests directives.

Name-based virtual hosts. Several <VirtualHost> blocks share the same IP and port, and the name in the request's Host header decides which site the request is for. This is the usual way to run many domains on one server.

Filters. Data going in and out of a connection passes through a chain of functions called "filters". The bottom filter that reads bytes from the socket is CORE_IN, and the bottom filter that writes to the socket is CORE. This post only looks at "which filter is added when"; how filters and buckets actually move data is the subject of Part 11.

1. Where the MPM hands over the socket

The path this post follows. Yellow is where structs (and their pools) are created; blue is where modules plug in through hooks.

As Part 8 showed, each MPM accepts connections in its own way. But the place where the accepted socket is handed on looks almost the same everywhere. Let us look at the simplest one, prefork.

        current_conn = ap_run_create_connection(ptrans, ap_server_conf, csd, my_child_num, sbh, bucket_alloc);
        if (current_conn) {
#if APR_HAS_THREADS
            current_conn->current_thread = thd;
#endif
            ap_process_connection(current_conn, csd);
            ap_lingering_close(current_conn);
        }

server/mpm/prefork/prefork.c lines 688–695

That is all there is: build a conn_rec through the create_connection hook, pass it to ap_process_connection(), and close the connection when it returns. #if APR_HAS_THREADS … #endif is a C preprocessor conditional: the line in between is compiled only if APR was built with thread support (Part 2). ptrans is the pool for this one connection, and csd is the accepted socket. The worker MPM goes through the same three steps (server/mpm/worker/worker.c lines 492–498).

ap_process_connection() is short.

AP_CORE_DECLARE(void) ap_process_connection(conn_rec *c, void *csd)
{
    ap_update_vhost_given_ip(c);

    ap_pre_connection(c, csd);

    if (!c->aborted) {
        ap_run_process_connection(c);
    }
}

server/connection.c lines 244–253

1. ap_update_vhost_given_ip() — uses the IP and port the connection came in on to pick a virtual host "for now". Section 6 covers it.
2. ap_pre_connection() — runs the pre_connection hook, which prepares the connection (socket options, filters). Section 2 covers it.
3. ap_run_process_connection() — hands the connection to whoever actually speaks the protocol on it. Section 3 covers it.

AP_CORE_DECLARE(void) is a macro around the return type void that adds whatever "exported function" marker the platform needs. When reading, just treat it as void.

Easy to get wrong: the event MPM does not call ap_process_connection()

If you grep for ap_process_connection(, you find it in prefork and worker but not in event, the default MPM. Event spells out the same three steps inside its own function process_socket(): ap_run_create_connection() (event.c line 1075) → ap_update_vhost_given_ip() (line 1099) → ap_pre_connection() (line 1101) → ap_run_process_connection() (line 1153). It spells them out because event handles a connection in several separate turns. Only a brand-new connection goes through the first three steps; a connection coming back for keep-alive starts straight at process_connection. The example in section 3 shows this.

2. Building the conn_rec and pre_connection — why the core stands last

The three connection hooks are created at the top of server/connection.c.

AP_IMPLEMENT_HOOK_RUN_FIRST(conn_rec *,create_connection,
                            (apr_pool_t *p, server_rec *server, apr_socket_t *csd, long conn_id, void *sbh, apr_bucket_alloc_t *alloc),
                            (p, server, csd, conn_id, sbh, alloc), NULL)
AP_IMPLEMENT_HOOK_RUN_FIRST(int,process_connection,(conn_rec *c),(c),DECLINED)
AP_IMPLEMENT_HOOK_RUN_ALL(int,pre_connection,(conn_rec *c, void *csd),(c, csd),OK,DECLINED)

server/connection.c lines 40–44

The core registers its own functions on the first two. The reason is written as a comment where they are registered.

    /* create_connection and pre_connection should always be hooked
     * APR_HOOK_REALLY_LAST by core to give other modules the opportunity
     * to install alternate network transports and stop other functions
     * from being run.
     */
    ap_hook_create_connection(core_create_conn, NULL, NULL,
                              APR_HOOK_REALLY_LAST);
    ap_hook_create_secondary_connection(core_create_secondary_conn, NULL, NULL,
                                        APR_HOOK_REALLY_LAST);
    ap_hook_pre_connection(core_pre_connection, NULL, NULL,
                           APR_HOOK_REALLY_LAST);

server/core.c lines 6148–6158

The core stands at the very end (REALLY_LAST) "to give other modules the opportunity to install alternate network transports and stop other functions from being run". A module that wants to treat the socket as something other than plain TCP gets called before the core and returns its own conn_rec (it is RUN_FIRST, so that ends it). If nobody steps in, the core does the default thing last.

core_create_conn — what goes into a conn_rec

Here is what core_create_conn() (server/core.c lines 5510–5597) does, in order.

1. It allocates a zero-filled conn_rec from the pool the MPM passed in (ptrans), creates a separate connection pool under it (tagged "master_conn") and stores it in c->pool.
2. It creates containers such as c->conn_config for per-module connection settings and the c->notes table.
3. From the socket it takes the server-side address (c->local_addr, c->local_ip) and the client address (c->client_addr, c->client_ip).
4. It puts the main server in c->base_server and the number given by the MPM in c->id.

There is no code here yet that attaches the socket to the conn_rec or adds filters. That is the next hook's job.

core_pre_connection — attach the socket and add the bottom filters

static int core_pre_connection(conn_rec *c, void *csd)
{
    conn_config_t *conn_config;
    apr_status_t rv;

    /* only the master connection talks to the network */
    if (c->master) {
        return DONE;
    }

    /* The Nagle algorithm says that we should delay sending partial
     * packets in hopes of getting more data.  We don't want to do
     * this; we are not telnet.  There are bad interactions between
     * persistent connections and Nagle's algorithm that have very severe
     * performance penalties.  (Failing to disable Nagle is not much of a
     * problem with simple HTTP.)
     */
    rv = apr_socket_opt_set(csd, APR_TCP_NODELAY, 1);
    /* ... */
    rv = apr_socket_timeout_set(csd, c->base_server->timeout);
    /* ... */
    conn_config = apr_pcalloc(c->pool, sizeof(*conn_config));
    conn_config->socket = csd;
    ap_set_core_module_config(c->conn_config, conn_config);

    ap_add_input_filter_handle(ap_core_input_filter_handle, NULL, NULL, c);
    ap_add_output_filter_handle(ap_core_output_filter_handle, NULL, NULL, c);

    return DONE;
}

server/core.c lines 5647–5694

• Turning off Nagle (APR_TCP_NODELAY). This disables the TCP feature that waits a little to bundle small packets together. The comment says "we are not telnet" and explains that this feature combined with keep-alive connections costs a lot of performance.
• Timeout. The socket gets the value of the Timeout directive (c->base_server->timeout).
• Keeping the socket. The socket is stored in the core module's per-connection config. That is what ap_get_conn_socket(c) fetches later.
• The two bottom filters. CORE_IN goes on the input side and CORE on the output side. Both are registered in the core's register_hooks() as AP_FTYPE_NETWORK, the lowest layer (server/core.c lines 6199–6207).

The return DONE at the end is worth a look too. pre_connection is RUN_ALL, so any value other than OK or DECLINED stops it — but the core is last anyway, so nothing is lost. ap_pre_connection(), which receives the result, treats both OK and DONE as success (server/core.c lines 5696–5718). If an earlier module returned an error, it marks the connection aborted, and if that kept the core from running so the socket was never attached, it calls core_pre_connection() itself once so the cleanup can still work.

Example: the filters are added at the very end of pre_connection

Let us check this with a small module that logs at each hook. It registers on pre_connection with APR_HOOK_MIDDLE (before the core) and on process_connection with APR_HOOK_FIRST, and prints the filters attached to the connection at that moment. Below are just those two functions from ex/u09-read-request/mod_u09_trace.c.

/* An example written for this post (not httpd's source) */
/* Once per connection: called before the core (REALLY_LAST) */
static int trace_pre_connection(conn_rec *c, void *csd)
{
    ap_log_cerror(APLOG_MARK, APLOG_NOTICE, 0, c,
                  "u09 pre_connection: id=%ld client_port=%d base_server=%s "
                  "in=[%s] out=[%s]",
                  c->id, c->client_addr->port, c->base_server->server_hostname,
                  chain(c->pool, c->input_filters),
                  chain(c->pool, c->output_filters));
    return OK;                      /* RUN_ALL: let the next module run */
}

/* process_connection is RUN_FIRST: return DECLINED and leave it to the http module */
static int trace_process_connection(conn_rec *c)
{
    ap_log_cerror(APLOG_MARK, APLOG_NOTICE, 0, c,
                  "u09 process_connection: id=%ld client_port=%d keepalives=%d "
                  "in=[%s] out=[%s]",
                  c->id, c->client_addr->port, c->keepalives,
                  chain(c->pool, c->input_filters),
                  chain(c->pool, c->output_filters));
    return DECLINED;
}

chain() is a small helper that walks a filter chain and joins the names with " > " (it is in the same file). This example was run in a Docker container (Ubuntu 24.04) with httpd 2.5.1-dev built as-is from the reference commit (e4af30fe6d). These are the first two lines of the error log after sending one request (id and client_port differ on every run).

[u09_trace:notice] u09 pre_connection: id=128 client_port=41194 base_server=a.example in=[(none)] out=[(none)]
[u09_trace:notice] u09 process_connection: id=128 client_port=41194 keepalives=0 in=[core_in] out=[core]

When our module's pre_connection runs there are no filters at all ((none)); by the very next hook, process_connection, core_in and core are attached. The only thing that ran in between is core_pre_connection() at the end of the hook.

Yellow boxes are the core, blue boxes the example module. The filter lists are exactly the two error log lines above.

Easy to get wrong

• The filter names in the log are lower case. The core registered "CORE_IN", yet the log says core_in. Filter names are lower-cased when they are registered (server/util_filter.c line 251).
• The core is not the only module that adds filters to a connection. mod_ssl, for example, is also registered on pre_connection with APR_HOOK_MIDDLE, so it runs before the core (modules/ssl/mod_ssl.c line 775). How TLS slips in at this point is covered in Part 15.

3. process_connection — where is the keep-alive loop?

The function that takes HTTP/1.x in the process_connection hook is registered by the http module.

    ap_hook_process_connection(ap_process_http_connection, NULL, NULL,
                               APR_HOOK_REALLY_LAST);

modules/http/http_core.c lines 358–359

REALLY_LAST again. It leaves room for modules that speak another protocol to step in first and take the connection. And they do: mod_http2 for HTTP/2 (modules/http2/h2_c1.c line 346) and mod_echo, which sends received bytes straight back (modules/echo/mod_echo.c line 206), both register on this hook. ap_process_http_connection() splits into two paths depending on the MPM.

static int ap_process_http_connection(conn_rec *c)
{
    if (async_mpm && !c->clogging_input_filters) {
        return ap_process_http_async_connection(c);
    }
    else {
        return ap_process_http_sync_connection(c);
    }
}

modules/http/http_core.c lines 247–255

async_mpm is set at startup by asking the MPM with ap_mpm_query(AP_MPMQ_IS_ASYNC, …) (line 346). The event MPM answers 1 (server/mpm/event/event.c lines 697–698). clogging_input_filters marks "a connection with an input filter that clogs the stream, so it is awkward to handle asynchronously".

Synchronous: prefork and worker loop inside this function

static int ap_process_http_sync_connection(conn_rec *c)
{
    /* ... */
    ap_update_child_status_from_conn(c->sbh, SERVER_BUSY_READ, c);
    while ((r = ap_read_request(c)) != NULL) {
        /* ... */
        if (r->status == HTTP_OK) {
            if (cs)
                cs->state = CONN_STATE_HANDLER;
            ap_update_child_status(c->sbh, SERVER_BUSY_WRITE, r);
            ap_process_request(r);
            /* ... */
            r = NULL;
        }

        if (c->keepalive != AP_CONN_KEEPALIVE || c->aborted)
            break;

        ap_update_child_status(c->sbh, SERVER_BUSY_KEEPALIVE, NULL);
        /* ... */
        apr_socket_opt_set(csd, APR_INCOMPLETE_READ, 1);
        apr_socket_timeout_set(csd, keep_alive_timeout);
        /* Go straight to select() to wait for the next request */
    }

    return OK;
}

modules/http/http_core.c lines 184–245

This is keep-alive exactly as we picture it. Read a request (ap_read_request), process it (ap_process_request, Part 10), and if the connection is to be kept (c->keepalive == AP_CONN_KEEPALIVE), switch the socket timeout to KeepAliveTimeout and wait for the next request. Meanwhile this process (or thread) is tied to this connection until the next request arrives. Whether to keep the connection is decided by ap_h1_set_keepalive() while the response is being built (modules/http/http_protocol.c lines 217–332); conditions such as reaching MaxKeepAliveRequests or the client sending Connection: close are gathered there.

Asynchronous: event reads one request and returns

static int ap_process_http_async_connection(conn_rec *c)
{
    /* ... */
    if (cs->state == CONN_STATE_PROCESSING) {
        ap_update_child_status_from_conn(c->sbh, SERVER_BUSY_READ, c);
        /* ... */
        if ((r = ap_read_request(c))) {
            if (r->status == HTTP_OK) {
                cs->state = CONN_STATE_HANDLER;
                /* ... */
                ap_update_child_status(c->sbh, SERVER_BUSY_WRITE, r);
                ap_process_async_request(r);
                /* ... */
                r = NULL;
            }

            if (cs->state != CONN_STATE_WRITE_COMPLETION &&
                cs->state != CONN_STATE_SUSPENDED) {
                /* Something went wrong; close the connection */
                cs->state = CONN_STATE_LINGER;
            }
        }
        else {   /* ap_read_request failed - client may have closed */
            cs->state = CONN_STATE_LINGER;
        }
    }

    return OK;
}

modules/http/http_core.c lines 139–182

There is no while. It reads and processes one request, records the connection's state (cs->state) and returns. When request processing finishes cleanly, ap_process_request_after_handler() sets the state to CONN_STATE_WRITE_COMPLETION (modules/http/http_request.c line 397). The event MPM sees that state, finishes sending whatever response is left, puts the socket on the list it watches and takes the thread back (see the comment at server/mpm/event/event.c lines 1161–1163). When data for the next request arrives, any free thread picks the connection up and calls the process_connection hook again. In other words, the keep-alive repetition lives in the MPM, not in the http module. The socket-watching side is in Part 8.

The keep-alive repetition is inside the http module's while on the right, and in the MPM on the left. Green marks the stretch where the thread lets go of the connection.

Example: two requests on one connection — event versus prefork

Give curl two URLs and it sends both requests, one after the other, over one connection. The access log prints the client port (%{remote}p) and how many keep-alive requests this connection has served before (%k). The same port means the same TCP connection.

curl -s -H 'Host: b.example' http://127.0.0.1:8080/index.txt http://127.0.0.1:8080/index.txt

With the event MPM (ids and ports differ on every run):

[u09_trace:notice] u09 pre_connection: id=142 client_port=41326 base_server=a.example in=[(none)] out=[(none)]
[u09_trace:notice] u09 process_connection: id=142 client_port=41326 keepalives=0 in=[core_in] out=[core]
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
[u09_trace:notice] u09 post_read_request: the_request="GET /index.txt HTTP/1.1" method=GET(0) uri=/index.txt args=(null) protocol=HTTP/1.1(1001) hostname=b.example server=b.example headers_in=3 conn_id=142 client_port=41326
[u09_trace:notice] u09 process_connection: id=143 client_port=41326 keepalives=1 in=[core_in] out=[core]
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
[u09_trace:notice] u09 post_read_request: the_request="GET /index.txt HTTP/1.1" method=GET(0) uri=/index.txt args=(null) protocol=HTTP/1.1(1001) hostname=b.example server=b.example headers_in=3 conn_id=143 client_port=41326
[u09_trace:notice] u09 process_connection: id=144 client_port=41326 keepalives=2 in=[core_in] out=[core]
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
--- access_log ---
port=41326 k=0 vhost=b.example "GET /index.txt HTTP/1.1" 200
port=41326 k=1 vhost=b.example "GET /index.txt HTTP/1.1" 200

The same configuration with only the MPM switched to prefork:

[u09_trace:notice] u09 pre_connection: id=0 client_port=41328 base_server=a.example in=[(none)] out=[(none)]
[u09_trace:notice] u09 process_connection: id=0 client_port=41328 keepalives=0 in=[core_in] out=[core]
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
[u09_trace:notice] u09 post_read_request: the_request="GET /index.txt HTTP/1.1" method=GET(0) uri=/index.txt args=(null) protocol=HTTP/1.1(1001) hostname=b.example server=b.example headers_in=3 conn_id=0 client_port=41328
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
[u09_trace:notice] u09 post_read_request: the_request="GET /index.txt HTTP/1.1" method=GET(0) uri=/index.txt args=(null) protocol=HTTP/1.1(1001) hostname=b.example server=b.example headers_in=3 conn_id=0 client_port=41328
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
--- access_log ---
port=41328 k=0 vhost=b.example "GET /index.txt HTTP/1.1" 200
port=41328 k=1 vhost=b.example "GET /index.txt HTTP/1.1" 200

In both cases the client port is the same, so there is one connection and one pre_connection. What differs is how often process_connection runs. Prefork calls it once, and the while inside reads both requests. Event calls it three times: once per request, plus a last time where it waited for another request, the client closed the connection and ap_read_request() returned NULL (the final pre_read_request has no post_read_request after it). For the same reason prefork also prints one extra pre_read_request at the end.

Easy to get wrong: in event, c->id changes

In the event log above, the id goes 142 → 143 → 144 even though it is the same connection. Event overwrites c->id with the number of whichever thread picked the connection up this time.

        /* Subsequent request on a conn, and thread number is part of ID */
        c->id = conn_id;

server/mpm/event/event.c lines 1130–1131

conn_id is computed with ID_FROM_CHILD_THREAD(child number, thread number) (line 1065), so it identifies "the thread handling this connection right now". Using c->id to tell whether two requests came on the same connection therefore gives the wrong answer under event. The client port together with %k, as above, is the safer check.

4. ap_read_request — in trunk, a filter reads the request line

Now the heart of this post, ap_read_request() (server/protocol.c lines 1306–1529). The function is long, so only the skeleton is shown.

AP_DECLARE(request_rec *) ap_read_request(conn_rec *conn)
{
    /* ... */
    request_rec *r = ap_create_request(conn);

    tmp_bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
    conn->keepalive = AP_CONN_UNKNOWN;

    ap_run_pre_read_request(r, conn);

    r->request_time = apr_time_now();
    rv = ap_get_brigade(r->proto_input_filters, tmp_bb, AP_MODE_READBYTES, APR_BLOCK_READ, 0);
    /* ... */
    for (e = APR_BRIGADE_FIRST(tmp_bb);
         e != APR_BRIGADE_SENTINEL(tmp_bb);
         e = APR_BUCKET_NEXT(e))
    {
        if (AP_BUCKET_IS_REQUEST(e)) {
            if (!breq) breq = e->data;
        }
        else if (AP_BUCKET_IS_ERROR(e)) {
            if (!berr) berr = e;
        }
        /* ... */
    }
    /* ... */
    if (breq) {
        /* ... */
        if (!ap_assign_request_line(r, method, uri, protocol)) {
            /* ... */
        }
    }

    if (berr) {
        access_status = ((ap_bucket_error *)(berr->data))->status;
        goto die_unusable_input;
    }
    /* ... */
    apply_server_config(r);
    /* ... */
    ap_add_input_filter_handle(ap_http_input_filter_handle,
                               NULL, r, r->connection);

    /* Validate Host/Expect headers and select vhost. */
    if (!ap_check_request_header(r)) {
        /* ... */
        goto die_before_hooks;
    }

    /* we may have switched to another server */
    apply_server_config(r);

    if ((access_status = ap_post_read_request(r))) {
        goto die;
    }
    /* ... */
    return r;
    /* ... */
}

server/protocol.c lines 1306–1529

Surprisingly, this function contains no code that reads lines from the socket. It calls ap_get_brigade() exactly once and then looks in the returned brigade (a list of data pieces called buckets, Part 11) for a REQUEST bucket or an ERROR bucket. Reading the request line and headers has already happened somewhere in between.

The filter does the reading (the grey box in the middle); ap_read_request() takes the resulting bucket and copies it into the request_rec.

Who reads it: the HTTP1_REQUEST_IN filter added in pre_read_request

The answer is in ap_run_pre_read_request(). The function the http module registers on this hook adds a filter for every request.

static void h1_pre_read_request(request_rec *r, conn_rec *c)
{
    if (!r->main && !r->prev
        && !strcmp(AP_PROTOCOL_HTTP1, ap_get_protocol(c))) {
        if (r->proxyreq == PROXYREQ_NONE) {
            ap_add_input_filter_handle(ap_h1_request_in_filter_handle,
                                       NULL, r, r->connection);
        }
        ap_add_output_filter_handle(ap_h1_response_out_filter_handle,
                                    NULL, r, r->connection);
    }
}

modules/http/http_core.c lines 274–285

If the connection speaks HTTP/1.x, it adds the HTTP1_REQUEST_IN filter on the input side (empty r->main and r->prev mean this is a "real" request, not a subrequest or an internal redirect; see Part 10). ap_get_brigade() then calls this filter, and the filter, ap_h1_request_in_filter() (modules/http/http_filters.c lines 2248–2355), works in two stages.

1. REQ_LINE — read_request_line() reads one request line, and ap_h1_tokenize_request_line() splits it into three pieces: method, address and protocol.
2. REQ_HEADERS — ap_get_mime_headers_core() reads headers until the empty line and puts them in the r->headers_in table.

If both succeed, it creates a REQUEST bucket holding the three pieces and the header table, puts it at the front of the brigade and returns. If anything fails, it puts in an ERROR bucket holding only a status code. On its first call the filter removes itself from the filter chain (ap_remove_input_filter(f), line 2272). The source comment only says it "needs to get out of the way of read_request_line()"; since ap_rgetline(), which reads the lines, reads through the same filter chain (r->proto_input_filters), leaving itself in place would mean going through itself again.

Back in ap_read_request(), the three pieces from the REQUEST bucket are passed to ap_assign_request_line() (server/protocol.c lines 1177–1304), which copies them into the fields of the request_rec. Next it adds the HTTP_IN filter to get ready for the body, calls ap_check_request_header() to check Host and pick the virtual host again (section 6), and finally runs the post_read_request hook.

Easy to get wrong: in 2.4.x, ap_read_request() reads the lines itself

In the 2.4.x branch, the one most of us install, ap_read_request() calls read_request_line() and ap_parse_request_line() inside the function, then calls ap_get_mime_headers_core() directly. 2.4.x has neither the HTTP1_REQUEST_IN filter nor the REQUEST bucket. Trunk separates "what HTTP means" from "how HTTP/1.x is written as text", and so moved the reading that is specific to HTTP/1.x into a filter. That is also why mod_http2, which handles HTTP/2, can produce the same REQUEST bucket (modules/http2/h2_request.c line 314). If you read the 2.4.x source alongside, remember that this part is built differently. The surviving ap_parse_request_line() is, in trunk, a short function that calls ap_h1_tokenize_request_line() and then ap_assign_request_line() (server/protocol.c lines 687–694).

What gets filled into request_rec

Gathering where the main fields of a request_rec are filled in while one request is read:

Here are the values the example module printed in post_read_request. The request sent is exactly the one from "Words to know first".

[u09_trace:notice] u09 post_read_request: the_request="GET /index.txt?x=1 HTTP/1.1" method=GET(0) uri=/index.txt args=x=1 protocol=HTTP/1.1(1001) hostname=b.example server=b.example headers_in=3 conn_id=128 client_port=41194

The part after ? was split off into args, and the three headers (Host, User-Agent, Connection) went into headers_in.

5. The request line and headers — limits and strictness

A request is data from outside, so the server has to decide "how long, and how loosely" it will accept it. These are the directives that set the rules, with their defaults.

The three LimitRequest* directives are RSRC_CONF, which means they may only appear in the server config and inside <VirtualHost> (server/core.c lines 4890–4896), and their values go into fields of server_rec such as limit_req_line (Part 7). The reason the default is 8190 is in a comment in httpd.h: internal buffers are two bytes larger than the limit, so a limit of 8190 gives an 8192-byte buffer.

You can see that + 2 where the request line is read.

    do {
        /* ensure ap_rgetline allocates memory each time thru the loop
         * if there are empty lines
         */
        ctx->request_line = NULL;
        len = 0;
        rv = ap_rgetline(&ctx->request_line, (apr_size_t)(ctx->r->server->limit_req_line + 2),
                         &len, ctx->r, strict ? AP_GETLINE_CRLF : 0, bb);

        if (rv != APR_SUCCESS) {
            return rv;
        }
        else if (len > 0) {
            /* got the line in ctx->r->the_request */
            return APR_SUCCESS;
        }
    } while (--num_blank_lines >= 0);
    /* too many blank lines */
    return APR_EINVAL;

modules/http/http_filters.c lines 2212–2230

Empty lines are skipped, but after 10 of them it gives up; if a single line is longer than the limit, ap_rgetline() returns APR_ENOSPC ("no space"). The filter turns these error codes into HTTP statuses.

            if ((rv = read_request_line(ctx, bb)) != APR_SUCCESS) {
                /* certain failures are answered with a HTTP error bucket
                 * and are terminal for parsing a request */
                ctx->method = ctx->uri = "-";
                ctx->protocol = "HTTP/1.0";
                if (APR_STATUS_IS_ENOSPC(rv)) {
                    http_status = HTTP_REQUEST_URI_TOO_LARGE;
                }
                else if (APR_STATUS_IS_TIMEUP(rv)) {
                    http_status = HTTP_REQUEST_TIME_OUT;
                }
                else if (APR_STATUS_IS_BADARG(rv)) {
                    http_status = HTTP_BAD_REQUEST;
                }
                else if (APR_STATUS_IS_EINVAL(rv)) {
                    http_status = HTTP_BAD_REQUEST;
                }
                goto cleanup;
            }

            if (!ap_h1_tokenize_request_line(r, ctx->request_line,
                                             &ctx->method, &ctx->uri, &ctx->protocol)) {
                http_status = HTTP_BAD_REQUEST;
                goto cleanup;
            }

modules/http/http_filters.c lines 2277–2301

A line that is too long gives 414, running out of time gives 408, and a malformed line gives 400. The header-count limit is checked by counting the headers one by one as they are read.

            if (r->server->limit_req_fields
                    && (++fields_read > r->server->limit_req_fields)) {
                r->status = HTTP_BAD_REQUEST;
                apr_table_setn(r->notes, "error-notes",
                               "The number of request header fields "
                               "exceeds this server's limit.");
                ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r, APLOGNO(00563)
                              "Number of request headers exceeds "
                              "LimitRequestFields");
                return;
            }

server/protocol.c lines 930–940

If limit_req_fields is 0, the left side of && is false and the check is skipped entirely. The "0 = unlimited" in the directive's description is this one line.

HttpProtocolOptions Strict — how picky to be

The default for HttpProtocolOptions is Strict. All over the code this setting is read into a variable called strict (conf->http_conformance != AP_HTTP_CONFORMANCE_UNSAFE). Of the things that change under strict, these are the ones confirmed in the source.

• Lines must end in CR LF. The AP_GETLINE_CRLF flag is set, so a line ending in a bare \n becomes APR_EINVAL and a 400 (server/protocol.c lines 393–402).
• Exactly one space between parts of the request line. Two or more spaces between method, address and protocol, or extra spaces at the end, give "Excess whitespace" and a 400 (tokenize_request_line(), modules/http/http_protocol.c lines 1713–1826).
• The protocol must be upper-case HTTP/x.y. Something like http/1.1 is fixed up and accepted under Unsafe, but is a 400 under Strict (r_assign_protocol(), server/protocol.c lines 1132–1175).
• Header names may only use HTTP token characters. A control character in a header value is also a 400 (the else /* Using strict RFC7230 parsing */ branch of ap_get_mime_headers_core()).

Example: 400 and 414

curl will not send a malformed request, so we send raw requests through bash's /dev/tcp (the container has no nc, which is why this approach was used). Here is a helper that shows only the first line of the response.

# Send a raw request as-is and show only the first line (the status line) of the response
raw() {
  exec 3<>/dev/tcp/127.0.0.1/8080
  printf '%b' "$1" >&3
  timeout 3 cat <&3 | head -1 | tr -d '\r'
  exec 3>&-
}

raw 'GET /index.txt HTTP/1.1 extra\r\nHost: b.example\r\n\r\n'
raw 'GET /index.txt http/1.1\r\nHost: b.example\r\n\r\n'
raw 'GET /index.txt HTTP/1.1\r\nConnection: close\r\n\r\n'

All three requests got HTTP/1.1 400 Bad Request. Below are the reasons logged with LogLevel core:debug http:debug, and the access log. The example module's u09 lines are left out of the error log (ports differ on every run).

[http:debug] AH03448: HTTP Request Line; Excess whitespace (disallowed by HttpProtocolOptions Strict)
[core:debug] AH10388: HTTP Request: Unrecognized protocol 'http/1.1' (perhaps whitespace was injected?)
[core:debug] AH00569: client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /index.txt
--- access_log ---
port=41234 k=0 vhost=a.example "-" 400
port=41238 k=0 vhost=a.example "GET /index.txt http/1.1" 400
port=41254 k=0 vhost=a.example "GET /index.txt HTTP/1.1" 400

Each of the three 400s is caught in a different place. The first in ap_h1_tokenize_request_line() inside the filter (the http module); the second in r_assign_protocol() after control is back in ap_read_request() (the core); the third after all reading is done, when ap_check_request_header() rejects an HTTP/1.1 request without Host (server/protocol.c lines 729–744).

Where the three 400s above and the 414 below are caught, in reading order. That headers are read inside the filter was checked in http_filters.c lines 2297–2307, and the later checks in protocol.c lines 1385 and 1466.

Next, a long request line. In the example configuration only the first virtual host, a.example, has LimitRequestLine 100; b.example keeps the default (8190). We sent a 150-character address with Host: b.example.

=== 3. long request line (Host: b.example, a.example has LimitRequestLine 100) ===
HTTP/1.1 414 Request-URI Too Long
--- error_log (u09/core/http) ---
[u09_trace:notice] u09 pre_connection: id=64 client_port=41286 base_server=a.example in=[(none)] out=[(none)]
[u09_trace:notice] u09 process_connection: id=64 client_port=41286 keepalives=0 in=[core_in] out=[core]
[u09_trace:notice] u09 pre_read_request: server=a.example limit_req_line=100
--- access_log ---
port=41286 k=0 vhost=a.example "-" 414

b.example's limit is 8190, yet we got a 414. The next "easy to get wrong" explains why.

Easy to get wrong

• The request-line limit comes from the connection's default virtual host, not from the one chosen by Host. When the request line is read, the Host header has not been read yet, so r->server is still the conn->base_server that ap_create_request() put there. The example's pre_read_request: server=a.example limit_req_line=100 is the value at that moment. The header size and count limits (r->server->limit_req_fieldsize and friends in ap_get_mime_headers_core()) are checked at the same point, so the same applies to them. Keep this in mind if you want different limits for different name-based virtual hosts.
• %r is sometimes logged as "-". When the filter fails, only an ERROR bucket comes back with no REQUEST bucket, so ap_assign_request_line() is never called and r->the_request stays empty. That is what happened with the first 400 and with the 414 above. The second 400 got as far as a successful split, so its request line made it into the log.
• The 414 has no "exceeds LimitRequestLine" log line (AH00565). That message only exists in the branch where ap_assign_request_line() fails inside ap_read_request() (server/protocol.c lines 1385–1396). At this commit, a line that is too long makes the filter return only an ERROR bucket, so that branch is never reached. In the example, core:debug was on, and the line still did not appear. I did not run 2.4.x to see whether it logs this message in the same situation.
• A request line with "extra" on the end is reported as "Excess whitespace". Under Strict, extra space after the protocol is flagged as a whitespace error before anything looks at what follows (the else if (strict && *ll) in tokenize_request_line()). Guessing the cause from the log message alone can mislead you.

6. Picking the virtual host, and post_read_request

It is chosen in two steps

A virtual host is not chosen in one go. When the connection arrives, only the IP and port are known; the Host name is known only after all request headers have been read. So httpd chooses in two steps.

The red box is the trap from section 5: everything read before step 2 follows the limits of the server chosen in step 1.

Step 1 is ap_update_vhost_given_ip(), which we met in section 1.

AP_DECLARE(void) ap_update_vhost_given_ip(conn_rec *conn)
{
    ipaddr_chain *trav;
    apr_port_t port;

    /* scan the hash table for an exact match first */
    trav = find_ipaddr(conn->local_addr);

    if (trav) {
        /* save the name_chain for later in case this is a name-vhost */
        conn->vhost_lookup_data = trav->names;
        conn->base_server = trav->server;
        return;
    }

    /* maybe there's a default server or wildcard name-based vhost
     * matching this port
     */
    port = conn->local_addr->port;

    trav = find_default_server(port);
    if (trav) {
        conn->vhost_lookup_data = trav->names;
        conn->base_server = trav->server;
        return;
    }

    /* otherwise we're stuck with just the main server
     * and no name-based vhosts
     */
    conn->vhost_lookup_data = NULL;
}

server/vhost.c lines 1267–1298

It looks up the server-side address the connection came in on (local_addr) in the "IP:port → virtual host" hash table (iphash_table) built while reading the configuration, and if that fails, in the list of wildcards like *:port (default_list). When it finds one, it records two things on the connection: base_server gets the first virtual host on that address, and vhost_lookup_data gets the list of names that share the address. A source comment notes that this table does not change after the configuration is read, so it is safe for many threads to read at once (server/vhost.c lines 87–94).

Step 2 is ap_update_vhost_from_headers_ex(), called by ap_check_request_header() once the headers are in (server/vhost.c lines 1160–1211). It cleans up the Host header (if the request line carries a host, as in http://name/…, that takes priority), stores it in r->hostname, and walks the name list saved in step 1 with update_server_from_aliases() (lines 986–1077). When it finds the first server whose ServerName or ServerAlias matches, it sets r->server = s. The comment in this function says that even if Host contains a port, that port is ignored and the socket's real port is used, and it gives five reasons (lines 988–1000).

If no name matches, update_server_from_aliases() returns HTTP_BAD_REQUEST, but that value is only turned into an actual 400 when StrictHostCheck On is set.

    strict_host_check = (conf->strict_host_check == AP_CORE_CONFIG_ON);
    access_status = ap_update_vhost_from_headers_ex(r, strict_host_check);
    if (strict_host_check && access_status != HTTP_OK) { 

server/protocol.c lines 708–710

Otherwise r->server stays base_server, that is, the first virtual host on that address. The familiar rule "with name-based virtual hosts, the first one listed is the default" shows up in the code like this.

Example: a different site depending on Host

On *:8080 we set up two virtual hosts, a.example (listed first) and b.example, each with a file containing "site A" or "site B" in its document folder. Here is the virtual host table from httpd -S.

VirtualHost configuration:
*:8080                 is a NameVirtualHost
         default server a.example (/tmp/u09.conf:20)
         port 8080 namevhost a.example (/tmp/u09.conf:20)
         port 8080 namevhost b.example (/tmp/u09.conf:25)

Three requests, changing only Host:

for h in a.example b.example c.example; do
  printf '%s -> ' "$h"; curl -s -H "Host: $h" http://127.0.0.1:8080/index.txt
done

a.example -> site A
b.example -> site B
c.example -> site A

c.example, which matches no name, went to a.example, the default server. The example module's log tells the same story. For every request, pre_read_request saw server=a.example; at post_read_request only the b.example request had changed to server=b.example, and the c.example request showed hostname=c.example server=a.example.

post_read_request — the first place to step in once the request is read

The last step of ap_read_request() is ap_post_read_request().

AP_DECLARE(int) ap_post_read_request(request_rec *r)
{
    int status;

    if ((status = ap_run_post_read_request(r))) {
        return status;
    }

    /* Enforce http(s) only scheme for non-forward-proxy requests */
    if (!r->proxyreq
            && r->parsed_uri.scheme
            && (ap_cstr_casecmpn(r->parsed_uri.scheme, "http", 4) != 0
| (r->parsed_uri.scheme[4] != '\0'
                    && (apr_tolower(r->parsed_uri.scheme[4]) != 's'
| r->parsed_uri.scheme[5] != '\0')))) {
        return HTTP_BAD_REQUEST;
    }

    return OK;
}

server/protocol.c lines 1531–1550

post_read_request is a RUN_ALL hook (lines 2554–2555). The request line and headers are all there and the virtual host is settled, but mapping the URL to a file has not started yet — so this is where "things you want to do first for every request" get hooked in.

• The http module's h1_post_read_request() (REALLY_FIRST) — if there is a Transfer-Encoding whose last coding is not chunked, it returns 400; if one arrives together with Content-Length, it removes Content-Length and marks the connection to be closed. It then adds the HTTP1_BODY_IN filter that will read the body (modules/http/http_core.c lines 287–332).
• mod_remoteip (APR_HOOK_FIRST, modules/metadata/mod_remoteip.c line 1317) — looks at headers like X-Forwarded-For sent by a front proxy and replaces r->useragent_ip with the real client address (line 768). It has to happen this early so that later access control and logging see the replaced address.

Once all hooks are done, a request that is not a proxy request but whose address has a scheme other than http or https is blocked with a 400. If it gets through all of this, ap_read_request() returns the finished request_rec, and the ap_process_request() from section 3 takes over.

Easy to get wrong: post_read_request is not called when there is an error

ap_read_request()'s failure exits are a tail of labels: die_unusable_input → die_before_hooks → die. When the header checks fail it jumps to die_before_hooks, skipping the post_read_request hook, and goes straight to ap_die() to send the error response. When the input itself is broken (die_unusable_input), it also strips the input filters stacked above CORE_IN and marks the connection to be closed (server/protocol.c lines 1487–1529). That is why the 400 and 414 requests in section 5 produced no post_read_request log lines at all. Code hooked into post_read_request should assume it only ever sees "well-formed requests".

What this post leaves out

To keep to the main thread, these are only named, with their locations.

• The create_secondary_connection hook for "secondary connections" inside one connection, as HTTP/2 uses them, and core_create_secondary_conn() (server/core.c lines 5599–5645).
• ap_lingering_close(), which reads and discards a little leftover input when closing (server/connection.c line 183 onwards), and event's handling of CONN_STATE_LINGER.
• The details of ap_fgetline_core(), which reads one line (server/protocol.c lines 213–526) — line folding (obs-fold), growing the buffer while reading, non-blocking reads.
• Handling of Expect: 100-continue (server/protocol.c lines 749–772), and HttpProtocolOptions RegisteredMethods and Require1.0.
• IP-based virtual hosts and ServerPath (check_serverpath(), server/vhost.c line 1080 onwards), and ap_fini_vhost_config(), which builds the hash table after the configuration is read (line 565 onwards).
• ap_check_pipeline(), which checks whether the next request is already waiting on the same connection (pipelining).

Summary

What's next

We followed one socket as it became a conn_rec, and its request line and headers as they became a request_rec. The request_rec now holds the method, the address, the headers and the chosen virtual host. But nobody has yet decided which file on disk /index.txt is, which <Directory> and <Location> settings apply to the request, or who will build the response. Part 10: Request Processing — From a URL to a File and a Handler follows the order of steps in ap_process_request() and ap_process_request_internal() that do that.

Compared with Spring: Spring MVC does not do this post's work itself. Accepting the socket, reading the request line and headers and building an HttpServletRequest is the job of a servlet container such as Tomcat; Spring's DispatcherServlet starts from that already-built request object (Spring Part 21). You can think of httpd's request_rec as sitting where HttpServletRequest sits.

• Part 3: The Big Picture — Core, APR, MPM, Modules and the Three Big Structs
• Part 6: Modules and Hooks — How a Module Plugs into the Server
• Part 8: MPMs — How prefork, worker and event Share Out Connections
• Part 11: Filters and Bucket Brigades — How a Response Flows Out