namalzi

← All posts

Digging into Apache httpd's Source 14: mod_proxy — Reverse Proxy and Load Balancer

Apache httpd · Published

We follow one ProxyPass line through the source until the request reaches another server. We see where the URL turns into a "proxy:..." r->filename, how proxy_handler() hands the work to the scheme handler hook, the worker and connection pool that each backend gets, how mod_proxy_http streams the response back through brigades, and how the byrequests balancer alternates between two backends — checked with two backends running in a container.

What this post covers

This is the fourteenth post in the "Digging into Apache httpd's Source" series. In Part 13 we watched a Require line being decided, and met providers: a module publishes a feature under a name, and another module looks it up by that name. This post looks at the case where httpd does not build the response itself but fetches it from another server — that is, mod_proxy. If you have ever put httpd in front of Tomcat or Node.js with a line like ProxyPass /app/ http://127.0.0.1:8080/, this is what happens behind that line.

We want to answer four questions.

Item Details
Target Apache httpd
Version 2.5.1-dev (trunk)
Commit e4af30fe6d (2026-10-24)
Files read most modules/proxy/mod_proxy.c, mod_proxy.h, proxy_util.c, mod_proxy_http.c, mod_proxy_balancer.c, balancers/mod_lbmethod_byrequests.c
APR read alongside APR-util 1.6.3 (tag 1.6.3, commit 28327eeca2) — misc/apr_reslist.c

All line numbers in this post refer to this commit. How to get the same source and build the Docker image for the examples is in Part 1. The big picture in this post is the same on the 2.4.x branch that most of us install — 2.4.x's mod_proxy.c hooks the same functions into the same hooks, mod_lbmethod_byrequests.c is exactly the same file on both branches (same git object hash), and mod_proxy_http.c handles the ws scheme there too (the line numbers differ).

The six files in modules/proxy/ above add up to more than 15,000 lines (proxy_util.c alone is 6,053). This post picks only the main path that a first-time reader can follow. What was left out is listed by name and location under "What this post leaves out" at the end.

Words to know first

A proxy is "a server that goes on your behalf". It takes a client's request, sends the same request to another server, and hands the answer back to the client. There are two kinds, depending on whose behalf it acts on.

The backend is the server behind a reverse proxy that actually produces the response. Source comments also call it the origin server.

A worker is proxy_worker, the struct mod_proxy uses to represent "one backend". It holds settings such as the address and timeouts, and a connection pool to that backend (a place that keeps already-open connections and lends them out). This is not the same thing as the MPM's "worker threads" (Part 8).

A balancer is proxy_balancer, a group of workers. For each request it picks one member of the group. The rule for picking is split out under the name lbmethod (load balancing method) and plugged in as a provider.

Hooks and RUN_FIRST were covered in Part 6 and Part 10. In short, a hook is a spot where "the functions registered here will be called one after another", and in RUN_FIRST mode the calls stop at the first answer that is not DECLINED ("not my job"). mod_proxy not only plugs into the core's hooks, it also opens hooks of its own and lets submodules (mod_proxy_http and others) plug into them. That structure is the heart of this post.

A bucket brigade is a line of buckets holding response data, which flows out through the filters with ap_pass_brigade() (Part 11). A proxy puts what it reads from the backend into such a line and lets it flow straight into the client-side filters.

1. Forward and reverse — the difference is one field, r->proxyreq

Forward proxy versus reverse proxy. In a forward proxy, the browser knows the destination and sends a full URL; httpd marks r->proxyreq as PROXYREQ_PROXY and sends it to any server on the internet. In a reverse proxy, the browser sends only a path; httpd marks it PROXYREQ_REVERSE and sends it to a fixed few backends. The same mod_proxy does both jobs. What tells them apart in the source is a single field of request_rec.

request_rec has a field called proxyreq, which holds one of four values (include/httpd.h lines 1181–1184). A normal request has PROXYREQ_NONE (0), a forward proxy request PROXYREQ_PROXY, a reverse proxy request PROXYREQ_REVERSE, and the fake request the proxy creates to read the backend's response gets PROXYREQ_RESPONSE (section 5).

A forward proxy request is recognised as soon as the request line is read. A browser that uses a forward proxy writes the full URL in the request line, like GET http://other-server/a HTTP/1.1, not just the path. mod_proxy hooks proxy_detect() into post_read_request (the spot right after all headers have been read, seen in Part 9) to look for this.

    if (conf->req && r->parsed_uri.scheme) {
        /* but it might be something vhosted */
        if (!r->parsed_uri.hostname
| ap_cstr_casecmp(r->parsed_uri.scheme, ap_http_scheme(r)) != 0
| !ap_matches_request_vhost(r, r->parsed_uri.hostname,
                                         (apr_port_t)(r->parsed_uri.port_str
                                                      ? r->parsed_uri.port
                                                      : ap_default_port(r)))) {
            r->proxyreq = PROXYREQ_PROXY;
            r->uri = r->unparsed_uri;
            r->filename = apr_pstrcat(r->pool, "proxy:", r->uri, NULL);
            r->handler = "proxy-server";
        }
    }

modules/proxy/mod_proxy.c lines 818–831

conf->req is ProxyRequests On from the config file. If it is on, the request line has a scheme (http:), and the address is not one of this server's own virtual hosts, the request is marked as a forward proxy request. The last condition exists because, as the comment ("vhosted") says, even a full URL may point at this very server.

A reverse proxy request cannot be recognised from the request line alone, because the browser sends only a path like /app/a. That is why the next section's work — comparing the path with the ProxyPass list from the config file — is needed.

Easy to get wrong: X-Forwarded-For is added only for reverse proxying

A reverse proxy adds headers like X-Forwarded-For to tell the backend "the original client was this" (the code is in section 5). But that code runs only when PROXYREQ_REVERSE == r->proxyreq. Adding it in a forward proxy would reveal the addresses of users inside a company to any server outside, and the source comment calls it a "potential privacy violation" (proxy_util.c lines 4801–4820). In example 6 below we see that a forward request really does arrive without these headers.

2. How one ProxyPass line becomes r->filename

While reading the config: one alias and one worker

Reading ProxyPass /one/ http://127.0.0.1:8081/ calls add_pass() (mod_proxy.c lines 2006–2206). It does two things.

  1. It creates an alias (struct proxy_alias) and puts it into the server config's aliases array (line 2119). fake gets /one/ and real gets http://127.0.0.1:8081/ (mod_proxy.h lines 134–140).
  2. If the target starts with balancer:// it defines a balancer, otherwise a worker (line 2140, lines 2173–2183). If a worker that can already handle this URL exists (one whose name matches the start of the URL — found with ap_proxy_get_worker_ex(), which we meet in section 3), it is shared instead of created again (log AH01145 "Sharing worker").

The general process by which config turns into C structs is in Part 7.

When a request arrives: taking it over in the translate_name hook

These are the functions mod_proxy hooks into the core's hooks.

    /* handler */
    ap_hook_handler(proxy_handler, NULL, NULL, APR_HOOK_FIRST);
    /* filename-to-URI translation */
    ap_hook_pre_translate_name(proxy_pre_translate_name, NULL, NULL,
                               APR_HOOK_MIDDLE);
    /* mod_rewrite has a say on the uri before proxy translation */
    ap_hook_translate_name(proxy_translate_name, aszModRewrite, NULL,
                           APR_HOOK_FIRST);
    /* walk <Proxy > entries and suppress default TRACE behavior */
    ap_hook_map_to_storage(proxy_map_location, NULL,NULL, APR_HOOK_FIRST);
    /* fixup after mod_rewrite so that a [P] URL from there gets fixed up */
    ap_hook_fixups(proxy_fixup, aszModRewrite, NULL, APR_HOOK_FIRST);
    /* post read_request handling */
    ap_hook_post_read_request(proxy_detect, NULL, NULL, APR_HOOK_FIRST);

modules/proxy/mod_proxy.c lines 3428–3441

Lay this over the request processing phases from Part 10. For a normal request, in the translate_name phase the core appends the URL to DocumentRoot to build a disk path. mod_proxy hooks proxy_translate_name() in ahead of that (APR_HOOK_FIRST, but after mod_rewrite.c), and when the path matches the ProxyPass list, it puts a made-up name instead of a disk path into r->filename. translate_name is a RUN_FIRST hook, so once it returns OK, the core's translation is never called.

The comparison is a simple loop over the server config's alias array, from the front (just before it, a single ProxyPass written inside <Location> is checked first — lines 1042–1051).

    /* long way - walk the list of aliases, find a match */
    for (i = 0; i < conf->aliases->nelts; i++) {
        ent = &((struct proxy_alias *)conf->aliases->elts)[i];
        enc = (ent->flags & PROXYPASS_MAP_ENCODED) != 0;
        if (!(pre_trans ^ enc)) {
            int rv = ap_proxy_trans_match(r, ent, dconf);
            if (rv != HTTP_CONTINUE) {
                return rv;
            }
        }
    }

modules/proxy/mod_proxy.c lines 1053–1063

When the prefix matches, ap_proxy_trans_match() builds "proxy:" + real + the rest of the path (line 937) and puts three marks on the request.

        r->filename = found;
        r->handler = "proxy-server";
        r->proxyreq = PROXYREQ_REVERSE;

modules/proxy/mod_proxy.c lines 960–962

For /one/hello.txt, r->filename becomes proxy:http://127.0.0.1:8081/hello.txt. An r->filename that starts with "proxy:" acts as the sign "this is a proxy request" in every phase that follows.

The path of one request through mod_proxy. In the translate_name hook it is compared with the ProxyPass list and r->filename, r->handler and r->proxyreq are set; the map_to_storage hook matches Proxy sections instead of files; the fixups hook tidies the URL; then proxy_handler in the handler hook calls the pre_request hook (the balancer picks a worker), the scheme_handler hook (mod_proxy_http talks to the backend) and the post_request hook in turn. The top four boxes are the core's hooks, the bottom three are hooks mod_proxy opens itself. The yellow box is the sign translate_name leaves behind.

The other two hooks look at the same sign.

Easy to get wrong

3. proxy_handler() and the scheme handler hook

Check the sign, strip off the scheme

In the handler phase proxy_handler() is called. It is hooked with APR_HOOK_FIRST, so it is asked before other handlers.

    /* is this for us? */
    if (!r->filename) {
        return DECLINED;
    }

    /* We may have forced the proxy handler via config or .htaccess */
    if (!r->proxyreq && r->handler && strncmp(r->handler, "proxy:", 6) == 0) {
        /* ... the case of SetHandler "proxy:..." ... */
    }
    else if (r->proxyreq && strncmp(r->filename, "proxy:", 6) == 0) {
        apr_table_unset(r->notes, "proxy-sethandler");
        rc = OK;
    }
    if (rc != OK) {
        return rc;
    }

    uri = r->filename + 6;
    p = strchr(uri, ':');
    /* ... */
    scheme = apr_pstrmemdup(r->pool, uri, p - uri);

modules/proxy/mod_proxy.c lines 1251–1286

r->filename + 6 is a common C trick. A string is the address of its first character, so adding 6 gives the address just past the six characters "proxy:" (nothing is copied). From there up to the next : is the scheme — http or balancer.

Pick a worker, then call the scheme handler

The main part is one do { ... } while loop. It is long because it exists to retry with another worker on failure, but the skeleton of one pass is two lines.

    do {
        char *url = uri;
        /* Try to obtain the most suitable worker */
        access_status = ap_proxy_pre_request(&worker, &balancer, r, conf, &url);
        /* ... */
        AP_PROXY_RUN(r, worker, conf, url, attempts);
        access_status = proxy_run_scheme_handler(r, worker, conf,
                                                 url, NULL, 0);
        /* ... */
    } while (!PROXY_WORKER_IS_USABLE(worker) &&
             max_attempts > attempts++);

modules/proxy/mod_proxy.c lines 1386–1528 (the ProxyRemote handling and the error branches in the middle are cut)

  1. ap_proxy_pre_request() — decides which worker takes this request (proxy_util.c lines 2647–2703). It first runs mod_proxy's pre_request hook; for a balancer:// URL, mod_proxy_balancer raises its hand here (section 6). If nobody does, it looks for the worker whose name overlaps the URL the longest (ap_proxy_get_worker_ex() lines 1871–1954, comment "longest match" at line 1929). If there is none either, it uses the default worker for forward or reverse proxying (conf->forward, conf->reverse).
  2. proxy_run_scheme_handler() — calls the module that will actually talk to the backend.

The scheme handler hook — where submodules line up

proxy_run_scheme_handler is a hook mod_proxy created itself. It is made with the same kind of macro as the core's hooks.

APR_IMPLEMENT_EXTERNAL_HOOK_RUN_FIRST(proxy, PROXY, int, scheme_handler,
                                     (request_rec *r, proxy_worker *worker,
                                      proxy_server_conf *conf,
                                      char *url, const char *proxyhost,
                                      apr_port_t proxyport),(r,worker,conf,
                                      url,proxyhost,proxyport),DECLINED)

modules/proxy/mod_proxy.c lines 3474–3479

It is RUN_FIRST, so the registered functions are called in turn until the first answer that is not DECLINED. The functions plugged into this hook belong to submodules such as mod_proxy_http, mod_proxy_ajp and mod_proxy_fcgi. Each looks at the URL's scheme and returns DECLINED if it is not theirs. Here is the mod_proxy_http side.

static void ap_proxy_http_register_hook(apr_pool_t *p)
{
    ap_hook_post_config(proxy_http_post_config, NULL, NULL, APR_HOOK_MIDDLE);
    proxy_hook_scheme_handler(proxy_http_handler, NULL, NULL, APR_HOOK_FIRST);
    proxy_hook_canon_handler(proxy_http_canon, NULL, NULL, APR_HOOK_FIRST);
    /* ... */
}

modules/proxy/mod_proxy_http.c lines 2212–2218

    scheme = get_url_scheme(&u, &is_ssl);
    /* ... */
    if (!scheme || u[0] != '/' || u[1] != '/' || u[2] == '\0') {
        ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, APLOGNO(01113)
                      "HTTP: declining URL %s", url);
        return DECLINED; /* only interested in HTTP, WS or FTP via proxy */
    }

modules/proxy/mod_proxy_http.c lines 1950–1960

get_url_scheme() recognises only four schemes: http, https, ws and wss (lines 31–71). Everything else gets DECLINED.

How the scheme handler hook works. When proxy_handler calls proxy_run_scheme_handler, the registered functions are called in turn. mod_proxy_http handles http, https, ws and wss and that ends it; otherwise it returns DECLINED and the next module (for example mod_proxy_ajp, mod_proxy_fcgi) gets its turn. If all return DECLINED, the result is a 500 with AH01144. The real order depends on where each module hooks in (APR_HOOK_FIRST and so on) and the load order. The order in the picture is one example.

Thanks to this split, the mod_proxy core knows nothing about HTTP. It only does the common work — picking a worker, retrying, recording status — and leaves the actual conversation to a module per scheme. To add a new protocol (uwsgi, say), you write a module that hooks one function into this hook (mod_proxy_uwsgi.c line 615).

Easy to get wrong: the scheme handler is a hook, not a provider

It looks like the providers of Part 13, but it works differently. A provider is looked up by name and used (ap_lookup_provider("proxylbmethod", "byrequests", ...)). A scheme handler has no name tag; everyone is asked in turn and the first to raise a hand takes it. mod_proxy uses both styles side by side — a hook for schemes, a provider for the balancing algorithm (lbmethod, section 6).

Example: without mod_proxy_http

The examples in this post were run with httpd 2.5.1-dev built unchanged from the reference commit (e4af30fe6d), in a Docker container (Ubuntu 24.04). The example folder is ex/u14-mod-proxy/; a single run.sh starts two backend httpd instances (8081, 8082) and a front proxy (8080) inside the container, then sends requests. This is the core of the front proxy's config.

# (1) pass straight through to one backend
ProxyPass        /one/ http://127.0.0.1:8081/
ProxyPassReverse /one/ http://127.0.0.1:8081/
# (2) pass through without ProxyPassReverse (to compare the Location header)
ProxyPass        /raw/ http://127.0.0.1:8082/
# (3) alternate between the two backends
<Proxy "balancer://pair">
    BalancerMember http://127.0.0.1:8081
    BalancerMember http://127.0.0.1:8082
</Proxy>
ProxyPass /lb/ balancer://pair/

The other examples come in later sections; here we look only at step 5. With the same config minus the LoadModule proxy_http_module ... line (and the balancer lines), we restarted and requested /one/hello.txt.

== 5. without mod_proxy_http
500
[warn] [proxy] AH01144: No protocol handler was valid for the URL /one/hello.txt (scheme 'http'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

ProxyPass was still there, so the translation from section 2 happened, but no function in the scheme handler hook was willing to take http, so the hook returned DECLINED and proxy_handler() turned it into a 500 (lines 1530–1538). The error message telling you to check that the submodules are loaded with LoadModule comes straight from this structure.

4. Workers and the connection pool — backend connections are reused

Three structs

struct proxy_worker {
    proxy_hashes    hash;       /* hash of worker name */
    unsigned int local_status;  /* status of per-process worker */
    proxy_conn_pool     *cp;    /* Connection pool to use */
    proxy_worker_shared   *s;   /* Shared data */
    proxy_balancer  *balancer;  /* which balancer am I in? */
    /* ... */
};

modules/proxy/mod_proxy.h lines 508–521

A worker has two parts.

struct proxy_conn_pool {
    apr_pool_t     *pool;     /* The pool used in constructor and destructor calls */
    apr_sockaddr_t *addr;     /* Preparsed remote address info */
    apr_reslist_t  *res;      /* Connection resource list */
    proxy_conn_rec *conn;     /* Single connection for prefork mpm */
    apr_pool_t     *dns_pool; /* The pool used for worker scoped DNS resolutions */
};

modules/proxy/mod_proxy.h lines 310–316

res is APR-util's resource list (apr_reslist_t). It is a general-purpose tool that "creates up to N expensive things, lends them out and takes them back", and what it lends here is a proxy_conn_rec, which represents one connection. With the prefork MPM, which has only one thread, there is just one connection (conn) instead of a list.

typedef struct {
    conn_rec     *connection;
    request_rec  *r;           /* Request record of the backend request
                                * that is used over the backend connection. */
    proxy_worker *worker;      /* Connection pool this connection belongs to */
    /* ... */
    apr_socket_t *sock;        /* Connection socket */
    /* ... */
    unsigned int close:1;      /* Close 'this' connection */
    /* ... */
} proxy_conn_rec;

modules/proxy/mod_proxy.h lines 274–302

Note that connection is the very same conn_rec we saw in Part 3. The struct that represents the connection to a client is reused as-is for the connection to a backend. So the data sent to a backend also goes through that conn_rec's filters to the socket (section 5).

How a worker and its connection pool fit together. There is one proxy_worker per backend URL, one copy per child process. s points to proxy_worker_shared, holding the name, status, lbfactor, lbstatus, hmax and so on; for balancer members it lives in shared memory. cp points to proxy_conn_pool, whose res is APR-util's apr_reslist holding at most hmax proxy_conn_rec entries. A proxy_conn_rec holds a socket and a conn_rec. Yellow boxes are storage. The proxy_conn_rec entries at the bottom are the connections that survive from one request to the next.

Borrowing and returning

The size of the list is set when a child process starts, as child_init() calls ap_proxy_initialize_worker() for each worker (mod_proxy.c lines 3344–3346). If you do not set it, the limit (hmax) is the number of threads in a child process (raised by mod_http2's worker count when mod_http2 is loaded, proxy_util.c lines 2318–2331). One connection per thread means nobody ever has to wait.

        if (max_threads > 1) {
            /* Default hmax is max_threads to scale with the load and never
             * wait for an idle connection to proceed.
             */
            if (worker->s->hmax == 0) {
                worker->s->hmax = max_threads;
            }

modules/proxy/proxy_util.c lines 2332–2338

While handling a request, a connection is borrowed with ap_proxy_acquire_connection().

    if (worker->s->hmax && worker->cp->res) {
        rv = apr_reslist_acquire(worker->cp->res, (void **)conn);
    }
    else {
        /* create the new connection if the previous was destroyed */
        if (!worker->cp->conn) {
            rv = connection_constructor((void **)conn, worker, worker->cp->pool);
        }
        else {
            *conn = worker->cp->conn;
            worker->cp->conn = NULL;
            rv = APR_SUCCESS;
        }
    }

modules/proxy/proxy_util.c lines 2831–2844

apr_reslist_acquire() hands out an idle resource if there is one, creates a new one if there is none and the limit has not been reached, and waits until someone returns one if the limit has been reached (APR-util misc/apr_reslist.c lines 330–412). When done, ap_proxy_release_connection() calls connection_cleanup() to put it back.

    /* determine if the connection should be cleared, closed or reused */
    if (!worker->s->is_address_reusable) {
        apr_pool_t *p = conn->pool;
        apr_pool_clear(p);
        conn = connection_make(p, worker);
    }
    else if (!conn->sock
| (conn->connection
                 && conn->connection->keepalive == AP_CONN_CLOSE)
| !ap_proxy_connection_reusable(conn)) {
        socket_cleanup(conn);
    }
    /* ... */
    if (worker->s->hmax && worker->cp->res) {
        conn->inreslist = 1;
        apr_reslist_release(worker->cp->res, (void *)conn);
    }
    else {
        worker->cp->conn = conn;
    }

modules/proxy/proxy_util.c lines 1670–1697

The proxy_conn_rec itself always goes back into the list. But if the backend said it would close the connection (AP_CONN_CLOSE), or something went wrong in this request and close was set, only its socket is cleaned up (socket_cleanup()) and it goes back as an empty shell. The next request that borrows that shell has no socket and connects anew. If it came back with a live socket, the next request reuses that connection. This is keep-alive to the backend.

Whether it is really still alive is checked by the borrower right before use. ap_proxy_check_connection() does this, and if the connection is alive it logs "reusing backend connection" at trace2 level (proxy_util.c lines 3726–3818, the log at line 3785).

Example: watching connections being reused

The connection pool is per child process, so the example's front proxy is pinned to a single child with StartServers 1 and ServerLimit 1 (ThreadsPerChild 25). The backends write the address and port of whoever connected to them (= the proxy) into their access logs.

LogFormat "b$n peer=%h:%{remote}p Host=%{Host}i XFF=%{X-Forwarded-For}i XFH=%{X-Forwarded-Host}i XFS=%{X-Forwarded-Server}i \"%r\"" fwd

(run.sh replaces b$n with the backend number.) Here are the backend logs after sending 3 requests to /one/ (the first lines of examples 1, 3 and 4), 6 to /lb/ (example 2) and 1 to /raw/ (the second line of example 4), all through the proxy. Port numbers differ on every run.

== backend 1 access_log
b1 peer=127.0.0.1:41900 Host=127.0.0.1:8081 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b1 peer=127.0.0.1:41902 Host=127.0.0.1:8081 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b1 peer=127.0.0.1:41902 Host=127.0.0.1:8081 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b1 peer=127.0.0.1:41902 Host=127.0.0.1:8081 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b1 peer=127.0.0.1:41900 Host=127.0.0.1:8081 XFF=203.0.113.7, 127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b1 peer=127.0.0.1:41900 Host=127.0.0.1:8081 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /sub HTTP/1.1"
== backend 2 access_log
b2 peer=127.0.0.1:49342 Host=127.0.0.1:8082 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b2 peer=127.0.0.1:49342 Host=127.0.0.1:8082 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b2 peer=127.0.0.1:49342 Host=127.0.0.1:8082 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /hello.txt HTTP/1.1"
b2 peer=127.0.0.1:49358 Host=127.0.0.1:8082 XFF=127.0.0.1 XFH=127.0.0.1:8080 XFS=localhost "GET /sub HTTP/1.1"

The six requests backend 1 received arrived over only two connections (ports 41900 and 41902). The front proxy's error log also has six lines saying the same connection was reused.

== front error_log: backend connection reuse
[trace2] [proxy] http: reusing backend connection 127.0.0.1:41902<>127.0.0.1:8081
[trace2] [proxy] http: reusing backend connection 127.0.0.1:49342<>127.0.0.1:8082
[trace2] [proxy] http: reusing backend connection 127.0.0.1:41902<>127.0.0.1:8081
[trace2] [proxy] http: reusing backend connection 127.0.0.1:49342<>127.0.0.1:8082
[trace2] [proxy] http: reusing backend connection 127.0.0.1:41900<>127.0.0.1:8081
[trace2] [proxy] http: reusing backend connection 127.0.0.1:41900<>127.0.0.1:8081

Easy to get wrong: why two connections to the same backend?

Backend 1 (8081) is the target of two workers: the worker for ProxyPass /one/ http://127.0.0.1:8081/ and the worker for BalancerMember http://127.0.0.1:8081. The first goes into the server config's workers array and the second into the balancer's workers array, separately (ap_proxy_define_worker_ex() proxy_util.c lines 2139–2160). Every worker has its own connection pool, so the connections are separate too. In the log, the /one/ requests (lines 1, 5 and 6) went over 41900 and the requests through the balancer (lines 2–4) over 41902. Backend 2's last line (/raw/, 49358) differs from the balancer's connection (49342) for the same reason. It is not "one backend server = one connection pool" but "one worker = one connection pool".

Two more things to remember.

5. mod_proxy_http — sending the request and streaming the response back

Six steps

proxy_http_handler() numbers its steps in source comments, from Step One to Step Six. Here is the middle part.

        /* Step Two: Make the Connection */
        if (ap_proxy_check_connection(scheme, backend, r->server, 1,
                                      PROXY_CHECK_CONN_EMPTY)
                && ap_proxy_connect_backend(scheme, backend, worker,
                                            r->server)) {
            /* ... */
            status = HTTP_SERVICE_UNAVAILABLE;
            break;
        }

        /* Step Three: Create conn_rec */
        if ((status = ap_proxy_connection_create_ex(scheme, backend, r)) != OK)
            break;
        req->origin = backend->connection;
        /* ... */
        /* Step Four: Send the Request
         * On the off-chance that we forced a 100-Continue as a
         * kinda HTTP ping test, allow for retries
         */
        status = ap_proxy_http_request(req);
        /* ... */
        /* Step Five: Receive the Response... Fall thru to cleanup */
        status = ap_proxy_http_process_response(req);

modules/proxy/mod_proxy_http.c lines 2127–2169

The order in which mod_proxy_http handles one request. Take a proxy_conn_rec from the pool; in Step One decide the address and port; in prefetch build the request line and headers as a brigade (X-Forwarded-* are added here). In Step Two, reuse the pooled connection if it is alive, otherwise open a new socket. Step Three prepares the conn_rec, Step Four sends through the backend connection's output filters, Step Five passes the brigades read from the backend to the client, and finally the connection goes back into the pool if reusable. The yellow boxes are where this touches section 4's connection pool.

  1. Borrow a connection — ap_proxy_acquire_connection() at the top of the function (line 1969).
  2. Step One — ap_proxy_determine_connection() decides the address and port to connect to (lines 2069–2075).
  3. Build the headers — ap_proxy_http_prefetch() calls ap_proxy_create_hdrbrgd() to build the request line and headers for the backend as a brigade (line 573). It also reads a little of the request body in advance.
  4. Step Two — if ap_proxy_check_connection() returns 0 (alive), ap_proxy_connect_backend() after the && is never called. This short condition is the fork between "reuse it" and "connect anew".
  5. Step Three — for a new socket, ap_create_connection() builds a conn_rec and the pre_connection hook is run to attach filters (proxy_util.c lines 4150 and 4204). The backend connection gets the same treatment a client connection got in Part 9.
  6. Step Four and Five — send and receive (below).
  7. Step Six — ap_proxy_release_connection() puts the connection back (lines 2181–2186).

Sending: into the backend connection's output filters

Sending finally comes down to one line in ap_proxy_pass_brigade().

    status = ap_pass_brigade(origin->output_filters, bb);

modules/proxy/proxy_util.c line 5157

origin is the backend conn_rec from Step Three. The same ap_pass_brigade() used to send responses to clients is pointed the other way and handed the backend's filter chain.

Receiving: pass on the brigades as they are read

To receive the response, mod_proxy_http creates a fake request_rec on top of the backend connection (make_fake_req(), lines 780–817). Section 1's PROXYREQ_RESPONSE is the mark on this fake request (lines 1133–1137). After the response headers are read, the body is moved by a single loop.

            do {
                apr_off_t readbytes;
                apr_status_t rv;

                rv = ap_get_brigade(backend->r->input_filters, bb,
                                    AP_MODE_READBYTES, mode,
                                    req->sconf->io_buffer_size);
                /* ... */
                /* try send what we read */
                if (ap_pass_brigade(r->output_filters, pass_bb) != APR_SUCCESS
| c->aborted) {
                    /* ... */
                }
                /* ... */
            } while (!finish);

modules/proxy/mod_proxy_http.c lines 1711–1869

It reads one chunk (up to io_buffer_size) from the backend connection's input filters, hands it to the client request's output filters, and repeats until the EOS bucket (end marker) appears. The whole response is never collected in memory before sending, so proxying a large file does not make memory grow much. It is also why the output filters of Part 11 (compression and the like) can step in as usual.

Easy to get wrong: the connection is returned before the last chunk is sent

When the loop meets EOS, it returns the backend connection with ap_proxy_release_connection() before sending to the client with ap_pass_brigade() (lines 1814–1846). The comment's reason is "so that the backend isn't left waiting for a slow client". That is why the buckets are first moved into the request pool with apr_bucket_setaside() — once the connection goes back, its memory belongs to the next user (Part 4's "pools decide lifetimes" applies here too).

Headers the backend receives: X-Forwarded-*

The backend is connected to the proxy, not the client, so it has no way to know who the original client was. So ap_proxy_create_hdrbrgd() adds three headers.

    if (dconf->add_forwarded_headers) {
        if (PROXYREQ_REVERSE == r->proxyreq) {
            /* Add X-Forwarded-For: so that the upstream has a chance to
             * determine, where the original request came from.
             */
            apr_table_mergen(r->headers_in, "X-Forwarded-For",
                             r->useragent_ip);

            /* Add X-Forwarded-Host: so that upstream knows what the
             * original request hostname was.
             */
            if (saved_host) {
                apr_table_mergen(r->headers_in, "X-Forwarded-Host",
                                 saved_host);
            }

            /* Add X-Forwarded-Server: so that upstream knows what the
             * name of this proxy server is (if there are more than one)
             * XXX: This duplicates Via: - do we strictly need it?
             */
            apr_table_mergen(r->headers_in, "X-Forwarded-Server",
                             r->server->server_hostname);
        }
    }

modules/proxy/proxy_util.c lines 4821–4844

add_forwarded_headers is the ProxyAddHeaders directive and is on (1) by default (mod_proxy.c line 1815). apr_table_mergen() does not overwrite an existing header of the same name; it appends with , . All of this shows in the backend log above.

In example 6, sent through a forward proxy, all three are missing (- is how the LogFormat writes "no such header"). Example 6 went through a separate front server (8090) with only ProxyRequests On and no ProxyPass — on the front server that has the /raw/ worker (http://127.0.0.1:8082/), that worker's name would have matched and it would have been used instead. Port numbers differ on every run.

== 6. forward proxy (curl -x)
hello from backend 2 (port 8082)
b2 peer=127.0.0.1:48462 Host=127.0.0.1:8082 XFF=- XFH=- XFS=- "GET /hello.txt HTTP/1.1"
[trace2] [proxy] *: found forward proxy worker for http://127.0.0.1:8082/hello.txt

The last line records that ap_proxy_pre_request() from section 3 found no worker with a matching name and used the default worker for forward proxying (lines 2666–2669).

Fixing response headers: ProxyPassReverse

There is a problem in the other direction too. If the backend sends a redirect to its own address, like Location: http://127.0.0.1:8082/sub/, the browser tries to go to the hidden backend directly and fails. process_proxy_header(), which copies the response headers one by one, rewrites only a few headers with fixed names.

    static const struct {
        const char *name;
        ap_proxy_header_reverse_map_fn func;
    } transform_hdrs[] = {
        { "Location", ap_proxy_location_reverse_map },
        { "Content-Location", ap_proxy_location_reverse_map },
        { "URI", ap_proxy_location_reverse_map },
        { "Destination", ap_proxy_location_reverse_map },
        { "Set-Cookie", ap_proxy_cookie_reverse_map },
        { NULL, NULL }
    };

modules/proxy/mod_proxy_http.c lines 825–835

It is a table pairing header names with function pointers (the C-style design from Part 2). ap_proxy_location_reverse_map() walks the list registered with ProxyPassReverse, and if a value starts with real (the backend address), it replaces that part with fake (the path on the proxy side) (proxy_util.c lines 886–995). In example 4 we requested the directory sub without a trailing slash so that the backend's mod_dir would send a redirect (Part 12).

== 4. Location header: with / without ProxyPassReverse
Location: http://127.0.0.1:8080/one/sub/
Location: http://127.0.0.1:8082/sub/

/one/, which has ProxyPassReverse, was rewritten to the proxy's address; /raw/, which does not, leaked the backend address as-is.

6. The balancer — how byrequests alternates

Swapping in a worker in the pre_request hook

balancer://pair is not a real address. mod_proxy_balancer hooks proxy_balancer_pre_request() into the pre_request hook that ap_proxy_pre_request() from section 3 runs (mod_proxy_balancer.c line 2069). If the URL does not start with balancer:// it returns DECLINED; if it does, it finds the balancer (lines 472–480), picks one member, and rewrites the URL to that member's real address. AH01172 in the example log is that moment.

== front error_log: balancer:// rewritten
[debug] [proxy_balancer] AH01172: balancer://pair: worker (http://127.0.0.1:8081) rewritten to http://127.0.0.1:8081/hello.txt
[debug] [proxy_balancer] AH01172: balancer://pair: worker (http://127.0.0.1:8082) rewritten to http://127.0.0.1:8082/hello.txt

The scheme handler that runs afterwards receives an ordinary http:// URL and a worker, without knowing a balancer was involved. Adding a balancer does not change a single line of mod_proxy_http.

The picking rule is a provider

The function that picks a member is finder, a function pointer in the lbmethod the balancer holds.

    candidate = (*balancer->lbmethod->finder)(balancer, r);

    if (candidate)
        candidate->s->elected++;

modules/proxy/mod_proxy_balancer.c lines 353–356

lbmethod is a proxy_balancer_method struct (mod_proxy.h lines 588–596), and each algorithm module registers one as a provider, as in Part 13.

static void register_hook(apr_pool_t *p)
{
    ap_register_provider(p, PROXY_LBMETHOD, "byrequests", "0", &byrequests);
    ap_hook_post_config(lbmethod_byrequests_post_config, NULL, NULL, APR_HOOK_MIDDLE);
}

modules/proxy/balancers/mod_lbmethod_byrequests.c lines 151–155

When a balancer is defined, "byrequests" is looked up by name and set as the default (proxy_util.c line 1237), and if you change it with something like ProxySet lbmethod=bytraffic, that name is looked up instead (mod_proxy.c lines 457–473). bybusyness, bytraffic and heartbeat sit in the same folder and work the same way.

The byrequests calculation

The whole algorithm is two functions, about ten lines.

static int is_best_byrequests(proxy_worker *current, proxy_worker *prev_best, void *baton)
{
    int *total_factor = (int *)baton;

    current->s->lbstatus += current->s->lbfactor;
    *total_factor += current->s->lbfactor;

    return (!prev_best || (current->s->lbstatus > prev_best->s->lbstatus));
}

modules/proxy/balancers/mod_lbmethod_byrequests.c lines 28–36

static proxy_worker *find_best_byrequests(proxy_balancer *balancer,
                                request_rec *r)
{
    int total_factor = 0;
    proxy_worker *worker = ap_proxy_balancer_get_best_worker_fn(balancer, r, is_best_byrequests, &total_factor);

    if (worker) {
        worker->s->lbstatus -= total_factor;
    }

    return worker;
}

modules/proxy/balancers/mod_lbmethod_byrequests.c lines 83–94

ap_proxy_balancer_get_best_worker_fn is a shared function published by mod_proxy. For each usable member it calls the is_best callback to ask "is this better than the best so far?" (proxy_util.c lines 1391–1505, the callback call at line 1466). Common rules such as skipping broken members and handling spare and standby members are dealt with there; the algorithm module only decides "is it better".

One choice made by byrequests. When a request arrives, lbfactor is added to lbstatus for every usable worker and also to total. The worker with the largest lbstatus is picked, the one seen first on a tie. Only the chosen worker has total subtracted from its lbstatus. The request goes there, elected is increased by one, and it repeats for the next request. The yellow boxes are where the lbstatus in shared memory is changed.

In words: lbfactor is "this member's share of the work", and lbstatus is "how far behind its share it is". Every time, everyone gets their share added, the member furthest behind is picked to do the work, and the total of the shares handed out this round is subtracted from that member. So the sum of lbstatus never changes (the source comment's "the sum of all lbstatus does not change", lines 50–51), and members with a bigger share are picked more often.

The default lbfactor is 100 (proxy_util.c line 2171), and writing loadfactor=2 stores 2 times 100, i.e. 200 (mod_proxy.c lines 107–116). With both members at 100, as in the example, the calculation goes like this.

Request After adding (8081, 8082) Member picked After subtracting (8081, 8082)
1 100, 100 8081 (on a tie, the one seen first) -100, 100
2 0, 200 8082 0, 0
3 100, 100 8081 -100, 100
4 0, 200 8082 0, 0

Example: watching it alternate

We requested /lb/hello.txt six times.

== 2. balancer://pair, 6 requests
hello from backend 1 (port 8081)
hello from backend 2 (port 8082)
hello from backend 1 (port 8081)
hello from backend 2 (port 8082)
hello from backend 1 (port 8081)
hello from backend 2 (port 8082)

The front proxy's error log (with LogLevel proxy_balancer:debug and proxy:trace2 turned on) records each pick and the lbstatus at that moment.

== front error_log: byrequests choices
[debug] [proxy] AH10123: proxy: byrequests selected worker "http://127.0.0.1:8081" : busy 0 : lbstatus 100
[debug] [proxy] AH10123: proxy: byrequests selected worker "http://127.0.0.1:8082" : busy 0 : lbstatus 200
[debug] [proxy] AH10123: proxy: byrequests selected worker "http://127.0.0.1:8081" : busy 0 : lbstatus 100
[debug] [proxy] AH10123: proxy: byrequests selected worker "http://127.0.0.1:8082" : busy 0 : lbstatus 200
[debug] [proxy] AH10123: proxy: byrequests selected worker "http://127.0.0.1:8081" : busy 0 : lbstatus 100
[debug] [proxy] AH10123: proxy: byrequests selected worker "http://127.0.0.1:8082" : busy 0 : lbstatus 200

This matches the "After adding" column of the table. The log line is written right after the shared function picks, before find_best_byrequests() subtracts total (proxy_util.c lines 1498–1502). That is why 100 and 200 alternate.

Easy to get wrong

What this post leaves out

To keep to the main path, many things are left with just a name.

Recap

Concept One-line summary
r->proxyreq The field that separates forward (PROXYREQ_PROXY), reverse (PROXYREQ_REVERSE) and backend response (PROXYREQ_RESPONSE)
proxy_detect() Marks full-URL requests as forward proxy requests in post_read_request
ProxyPass Creates an alias and a worker at config time; at request time takes over in translate_name with r->filename = "proxy:…". Compared in written order, first match wins
The "proxy:" sign Every later hook — map_to_storage (<Proxy> walk), fixups, handler — looks at it
proxy_handler() The common frame that picks a worker (pre_request) and calls the scheme handler hook
Scheme handler hook A RUN_FIRST hook opened by mod_proxy. mod_proxy_http takes only http, https, ws and wss and declines the rest
proxy_worker One backend URL. Split into values to share (s) and a per-process connection pool (cp)
Connection pool APR-util's apr_reslist. Default limit is the child's thread count. A live socket is reused by the next request
mod_proxy_http Builds the headers as a brigade and sends them through the backend conn_rec's filters; streams the response with ap_get_brigade → ap_pass_brigade
X-Forwarded-* Only for reverse proxying; appended if already present
ProxyPassReverse Replaces the backend address with the proxy path in Location and similar response headers
Balancer Picks a member in the pre_request hook and swaps in its URL. The picking rule (lbmethod) is a provider
byrequests Add each member's share (lbfactor), pick the one furthest behind, subtract the total of the shares

What to look at next

In this post we only followed an https:// backend up to the point where mod_proxy_http sets backend->is_ssl and the Step Three connection setup code (proxy_util.c) asks, through ap_proxy_ssl_engine(), for TLS to be turned on for the connection. The side that receives that request — how TLS plugs into client and backend connections as filters — is the topic of the next post. It continues in Part 15: mod_ssl — Where TLS Plugs In.

Compared with the Spring series, two places look alike. The borrow/return pair for connections (ap_proxy_acquire_connection() / ap_proxy_release_connection()) sits in the same spot as DataSourceUtils.getConnection() / releaseConnection() in Spring Part 18. The difference is that Spring leaves the pool itself to the DataSource implementation, while httpd keeps an apr_reslist inside each worker. And mod_proxy_http is a server and an HTTP client at the same time; on the Spring side, the sending part is handled by WebClient, in section 5 of Spring Part 24.

How nginx runs the same reverse proxy on its event loop, with no worker threads, is set side by side with this post in Digging into nginx's Source 11: Upstream and Proxy.

More posts