Digging into nginx's Source 12: Load Balancing, Upstream Keepalive and the Stream Module — and Wrapping Up the Series

What this post covers

This is the twelfth and final post in the "Digging into nginx's Source" series. In Part 11 we followed how a single proxy_pass line installs a content handler, and how ngx_http_upstream_t connects to a backend, sends the request, receives the response and streams it back to the client. That post deliberately skipped one question — who picks the server to connect to? This post is the answer.

Four questions:

• When upstream {} lists several servers, which one does each request get? What exactly does weight=5 do?
• When a server dies, how does nginx notice, and when does it try that server again?
• Does nginx open a new backend connection for every request? If it reuses them, where does it keep them — and which default changed in this version?
• What do stream {} and http {} have in common, and where do they differ?

All line numbers in this post refer to this commit. Part 1 shows how to check out the same commit and build the Docker image used for the examples. The load-balancing code alone (ngx_http_upstream_round_robin.c plus eight modules/ngx_http_upstream_*_module.c files) is over 7,000 lines, and src/stream/ is over 25,000. This post sticks to the main path; what I left out is listed by name and location in "What I left out" at the end.

On the map from Part 2, this post zooms into the very last box of area C. One HTTP request, load balancing · keepalive · stream.

When Part 11's upstream asks "where should I connect?", this box answers.

Terms to know first

Upstream and peer. An upstream is "a server behind nginx" from Part 11, and the upstream {} block in the config is a group of them. The source calls each server in that group (strictly, each address) a peer. If server backend.example:8080; resolves to two addresses, you get two peers.

Load balancing is picking one peer for each request. nginx separates "picking" and "giving back" into two function pointers (get and free), and each balancing method (round robin, least_conn, ip_hash …) is a module that fills those pointers. Plugging modules in through tables of function pointers is the same idea we saw in Part 5.

Smooth weighted round robin. A weighted round robin that, given weights 5:1:1, doesn't send a a a a a b c in a clump but interleaves them as a a b a c a a. Section 2 walks through the numbers.

Keepalive (upstream side). Client-side keep-alive (Part 8) reuses the connection between the browser and nginx for the next request; upstream keepalive reuses the connection between nginx and a backend. It saves the cost of a new TCP connection (the 3-way handshake, plus the TLS handshake if there is one). The place that keeps finished connections open instead of closing them is a connection cache (connection pool).

Stream module. A TCP/UDP proxy enabled with a stream {} block. It doesn't parse HTTP; it moves bytes as they are. You use it to put nginx in front of MySQL or Redis.

Event-loop vocabulary such as posted events and the connection struct was covered in Part 7, and phases and checkers in Part 9. They come back when we look at the stream module.

1. The load-balancing frame — init_upstream, init, get, free

From a config block to four function pointers

upstream app {
    least_conn;                        # round robin if omitted
    server 127.0.0.1:9001 weight=5;
    server 127.0.0.1:9002;
}

This block becomes one ngx_http_upstream_srv_conf_t in the source, and the load-balancing handle sits at its very front.

typedef struct {
    ngx_http_upstream_init_pt        init_upstream;
    ngx_http_upstream_init_peer_pt   init;
    void                            *data;
} ngx_http_upstream_peer_t;

src/http/ngx_http_upstream.h lines 94–98

The two function pointers are called at different times.

• init_upstream — once, while reading the config. It looks at the server list and builds the peer array. A directive like least_conn; does little more than store its own function in this pointer (ngx_http_upstream_least_conn_module.c line 330).
• init — once per request. It creates a small per-request state struct and fills in the get/free pointers of the next subsection.

If init_upstream is empty — that is, if you wrote none of least_conn, ip_hash, hash, random — round robin is used. After the whole http {} block has been read, the upstream module's init_main_conf calls this for every block:

    for (i = 0; i < umcf->upstreams.nelts; i++) {

        init = uscfp[i]->peer.init_upstream ? uscfp[i]->peer.init_upstream:
                                            ngx_http_upstream_init_round_robin;

        if (init(cf, uscfp[i]) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }

src/http/ngx_http_upstream.c lines 7362–7370

The first line of ngx_http_upstream_init_round_robin() then sets the per-request function — us->peer.init = ngx_http_upstream_init_round_robin_peer; (ngx_http_upstream_round_robin.c line 51). least_conn calls this same round-robin initializer first and then replaces only peer.init (lines 65–79). In other words, building the peer array is shared with round robin; only the picking rule differs.

Per request: pick with get, give back with free

The per-request side is ngx_peer_connection_t. The u->peer we saw in Part 11 is this struct, and it lives in the event core (src/event/) — a sign that it isn't HTTP-specific (in section 5 the stream module uses it too).

struct ngx_peer_connection_s {
    ngx_connection_t                *connection;

    struct sockaddr                 *sockaddr;
    socklen_t                        socklen;
    ngx_str_t                       *name;

    ngx_uint_t                       tries;
    ngx_msec_t                       start_time;

    ngx_event_get_peer_pt            get;
    ngx_event_free_peer_pt           free;
    ngx_event_notify_peer_pt         notify;
    void                            *data;
    /* ... */
    unsigned                         cached:1;
    /* ... */
};

src/event/ngx_event_connect.h lines 36–79 (cuts marked)

get picks a peer and fills in sockaddr and name; free reports success or failure when the request is done with that peer. data is a void * holding whatever state the balancing method needs, and tries is "how many more times we may retry on another server." They are called from these places:

Yellow is state sitting in memory; blue is a handle holding function pointers. The config part (bottom) is built once; the per-request part (top) is built for every request.

Round robin's per-request initializer, ngx_http_upstream_init_round_robin_peer(), builds the per-request half of this picture. It allocates rrp from the request pool, prepares the tried bitmap, and stores the round-robin functions in r->upstream->peer.get and peer.free (lines 521–565). tried is a bitmap with one bit per peer marking "this request has already tried it." With 64 servers or fewer (8 * sizeof(uintptr_t) = 64 on a 64-bit machine) it doesn't allocate anything and uses a single uintptr_t inside the struct as the bitmap. It keeps a failed-over request from picking the same server again.

Easy to get wrong: peer state is per worker

The peer array is created in the config pool with ngx_pcalloc(cf->pool, ...) (lines 145–154). The master builds the config pool and each worker gets a copy through fork() (Part 6), so current_weight, fails and conns are counted separately in each worker. With four workers, the "5:1:1" order and the max_fails failure counts run independently in each one. That's why the examples below run with worker_processes 1;. To make workers share one state, the zone directive moves the peers into shared memory (ngx_http_upstream_zone_copy_peers() in ngx_http_upstream_zone_module.c). Only then do macros like ngx_http_upstream_rr_peers_wlock() in the header actually lock — without shared memory peers->shpool is empty and nothing is locked (round_robin.h lines 141–145).

2. Smooth weighted round robin — spreading 5:1:1 evenly

Three numbers: weight, effective_weight, current_weight

The peer struct carries three weights.

struct ngx_http_upstream_rr_peer_s {
    struct sockaddr                *sockaddr;
    socklen_t                       socklen;
    ngx_str_t                       name;
    ngx_str_t                       server;

    ngx_int_t                       current_weight;
    ngx_int_t                       effective_weight;
    ngx_int_t                       weight;

    ngx_uint_t                      conns;
    ngx_uint_t                      max_conns;

    ngx_uint_t                      fails;
    time_t                          accessed;
    time_t                          checked;

    ngx_uint_t                      max_fails;
    time_t                          fail_timeout;
    /* ... */
    ngx_uint_t                      down;
    /* ... */
    ngx_http_upstream_rr_peer_t    *next;
    /* ... */
};

src/http/ngx_http_upstream_round_robin.h lines 47–105 (cuts marked)

• weight — the value from the config (weight=5). It never changes.
• effective_weight — how much we currently trust the peer. It starts equal to weight, drops on failure (section 3), and recovers by 1 on every pick.
• current_weight — a working value that rises and falls on every pick. It starts at 0.

If a server line sets nothing, the defaults are weight=1, max_fails=1 and fail_timeout=10 (seconds).

    weight = 1;
    max_conns = 0;
    max_fails = 1;
    fail_timeout = 10;

src/http/ngx_http_upstream.c lines 6546–6549

The picking function

With a single server (peers->single) there's nothing to compute and that server is returned; with several, ngx_http_upstream_get_peer() picks.

    for (peer = rrp->peers->peer, i = 0;
         peer;
         peer = peer->next, i++)
    {
        n = i / (8 * sizeof(uintptr_t));
        m = (uintptr_t) 1 << i % (8 * sizeof(uintptr_t));

        if (rrp->tried[n] & m) {
            continue;
        }

        if (peer->down) {
            continue;
        }

        if (peer->max_fails
            && peer->fails >= peer->max_fails
            && now - peer->checked <= peer->fail_timeout)
        {
            continue;
        }

        if (peer->max_conns && peer->conns >= peer->max_conns) {
            continue;
        }

        peer->current_weight += peer->effective_weight;
        total += peer->effective_weight;

        if (peer->effective_weight < peer->weight) {
            peer->effective_weight++;
        }

        if (best == NULL || peer->current_weight > best->current_weight) {
            best = peer;
            p = i;
        }
    }

    /* ... */

    if (best == NULL) {
        return NULL;
    }

    best->current_weight -= total;

src/http/ngx_http_upstream_round_robin.c lines 858–910 (the /* ... */ is the #if (NGX_HTTP_UPSTREAM_SID) block for sticky — compiled into our build, but just passed through when sticky isn't used)

After picking, it records the choice in rrp->current, sets its tried bit, and — if fail_timeout has passed — stamps a fresh time with best->checked = now; (lines 916–926). That last line comes back in section 3.

Four filters (the red branches) and three weight steps. The final line that subtracts total from the chosen server is the secret of the "smooth" spreading.

The calculation has three steps: ① add each live peer's weight to its current_weight; ② pick the largest; ③ subtract the sum of weights (total) from the one you picked. After one full round (as many requests as the weight sum), every current_weight is back at 0. Here is 5:1:1 (sum 7) by hand, calling the servers a, b and c:

At request 3, b wins the 3-vs-3 tie with c because the comparison is > (an equal value doesn't replace the current best). The heavy server a is picked five times, but not five in a row — b and c slip in between. The point is to avoid bursts of requests piling onto one backend.

Example 1: weights 5:1:1, and one dead server

This example ran on nginx 1.31.9 built straight from the reference commit (9a7439fd79) inside a Docker container (Debian bookworm, --with-debug). The example folder is ex/u12-weights. The backends are three tiny python3 servers (9001–9003) that print their own port.

# Example written for this post (not part of nginx's source)
worker_processes 1;
error_log /tmp/u12w/error.log debug;

http {
    # 1) weights 5:1:1
    upstream weighted {
        server 127.0.0.1:9001 weight=5;
        server 127.0.0.1:9002;
        server 127.0.0.1:9003;
    }

    # 2) nothing listens on 9009 (max_fails=1, fail_timeout=10s are the defaults)
    upstream withdead {
        server 127.0.0.1:9001;
        server 127.0.0.1:9002;
        server 127.0.0.1:9009;
    }

    server {
        listen 8080;
        location /w/ { proxy_pass http://weighted/; }
        location /d/ { proxy_pass http://withdead/; }
    }
}

(I omitted the second explanatory comment line and the pid, events {} and access_log lines from the real file, which also writes the location blocks over several lines.) Here are the ports of the backends that answered 14 requests to /w/, and the get rr peer, current: lines of the first seven requests from the debug log.

=== 1. weight=5,1,1 : 14 requests ===
9001 9001 9002 9001 9003 9001 9001 9001 9001 9002 9001 9003 9001 9001 
=== 1b. current_weight of the chosen peer, first 7 (debug log) ===
33#33: *2 get rr peer, current: 000055F74EDCB0C0 -2
33#33: *4 get rr peer, current: 000055F74EDCB0C0 -4
33#33: *5 get rr peer, current: 000055F74EDCB1C8 -4
33#33: *7 get rr peer, current: 000055F74EDCB0C0 -1
33#33: *8 get rr peer, current: 000055F74EDCB2D0 -2
33#33: *10 get rr peer, current: 000055F74EDCB0C0 2
33#33: *11 get rr peer, current: 000055F74EDCB0C0 0

The order matches the table (a a b a c a a, then from the top again), and the numbers in the log match, one by one, the chosen server's value in column ③ (−2, −4, b's −4, −1, c's −2, 2, 0). The ngx_log_debug2 at lines 753–755 prints this line after best->current_weight -= total. The worker pid (33#33), pointer addresses and connection numbers *N differ on every run (the connection numbers skip by one because the connections nginx opens to backends also take numbers).

Six requests to /d/ in the same run:

=== 2. one dead server (9009): 6 requests ===
200 9001
200 9002
200 9001
200 9002
200 9001
200 9002

The client never saw a failure. The third request picked 9009, was refused, and moved on to another server. We'll read that request's log in section 3.

3. When a server dies — max_fails and fail_timeout

When a connection is refused or times out, ngx_http_upstream_next() from Part 11 calls free with NGX_PEER_FAILED. Round robin's free counts it like this:

    if (state & NGX_PEER_FAILED) {
        now = ngx_time();

        peer->fails++;
        peer->accessed = now;
        peer->checked = now;

        if (peer->max_fails) {
            peer->effective_weight -= peer->weight / peer->max_fails;

            if (peer->fails >= peer->max_fails) {
                ngx_log_error(NGX_LOG_WARN, pc->log, 0,
                              "upstream server temporarily disabled");
            }
        }

        ngx_log_debug2(NGX_LOG_DEBUG_HTTP, pc->log, 0,
                       "free rr peer failed: %p %i",
                       peer, peer->effective_weight);

        if (peer->effective_weight < 0) {
            peer->effective_weight = 0;
        }

    } else {

        /* mark peer live if check passed */

        if (peer->accessed < peer->checked) {
            peer->fails = 0;
        }
    }

src/http/ngx_http_upstream_round_robin.c lines 1056–1087

Two things happen here.

1. Once the failure count (fails) reaches max_fails, the loop in section 2 skips the peer while now - peer->checked <= peer->fail_timeout. With the defaults, one failure takes the server out for 10 seconds.
2. The weight (effective_weight) drops by weight / max_fails. Even after the server comes back it doesn't get its full share at once; it recovers by 1 on each pick (the effective_weight++ in section 2).

The condition for resetting fails to 0 deserves a close look. A success doesn't clear it immediately; it clears only when accessed < checked. A failure sets accessed and checked to the same time, and checked gets a new time at the end of section 2's loop only when the peer is picked after fail_timeout has passed. So a success inside fail_timeout doesn't erase the failure count; the peer returns to normal only when it is picked again after fail_timeout and succeeds.

This source has no health-check module that probes backends on its own (the request struct has a health_check bit, but no code sets it). State changes only through the record of real requests failing and succeeding.

Here is the third request (*23) of Example 1 meeting 9009 — the warn/error lines of error.log, plus only that request's lines from the debug log.

=== 2b. error.log (warn and above) ===
[error] 33#33: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /d/ HTTP/1.1", upstream: "http://127.0.0.1:9009/", host: "127.0.0.1:8080"
[warn] 33#33: *23 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, server: , request: "GET /d/ HTTP/1.1", upstream: "http://127.0.0.1:9009/", host: "127.0.0.1:8080"
=== 2c. debug: the request that hit 9009 ===
33#33: *23 get rr peer, try: 3
33#33: *23 get rr peer, current: 000055F74EDCB6A8 0
33#33: *23 connect to 127.0.0.1:9009, fd:15 #24
33#33: *23 http next upstream, 2
33#33: *23 free rr peer 3 4
33#33: *23 free rr peer failed: 000055F74EDCB6A8 0
33#33: *23 get rr peer, try: 2
33#33: *23 get rr peer, current: 000055F74EDCB498 -1
33#33: *23 free rr peer 2 0

How to read it:

• try: 3 — this upstream's tries is the number of servers not marked down (3): peers->tries = t at lines 156–160.
• http next upstream, 2 — 2 is NGX_HTTP_UPSTREAM_FT_ERROR.
• free rr peer 3 4 — 3 tries left, state 4 = NGX_PEER_FAILED (ngx_event_connect.h line 19). With max_fails=1 the warn line appears at once, and effective_weight became 1 − 1/1 = 0.
• The second get picks again with try: 2. 9009 is out of the running both because its tried bit is set and because it is skipped. The final free rr peer 2 0 is a success (0).
• The next three requests (4th–6th) alternated between 9001 and 9002 only — for 10 seconds, 9009 trips the third filter in section 2's loop.

Easy to get wrong: with a single server, max_fails does nothing

At the top of the free function there is a separate branch for a single server (if (rrp->peers->single)) (lines 1038–1054). Failure or success, that branch resets fails to 0 and ends with pc->tries = 0;. With no other server to pick, taking this one out would be pointless. Write max_fails=3 on a one-server upstream and you will never see "temporarily disabled." (If there is a backup server, single becomes 0 and this branch isn't taken — lines 294–296.)

Other methods: a quick look at least_conn

least_conn's get has the same loop and filters as section 2; only the criterion changes to "smallest connections ÷ weight." It compares by multiplying instead of dividing.

        if (best == NULL
| peer->conns * best->weight < best->conns * peer->weight)
        {
            best = peer;
            many = 0;
            p = i;

        } else if (peer->conns * best->weight == best->conns * peer->weight) {
            many = 1;
        }

src/http/modules/ngx_http_upstream_least_conn_module.c lines 181–190

When several tie (many), it runs section 2's smooth weighted round robin among them (lines 200–248). conns is the value that round robin's get increments (peer->conns++) and free decrements (peer->conns--) — counted per worker (section 1), so with several workers it means more when combined with zone. ip_hash (for IPv4, a hash of the first three bytes of the client address), hash (any key; consistent makes it a ring hash), random, and least_time, new in 1.31.x (absent from 1.30.x), all put their own function in init_upstream and only use a different get on top of round robin's peer array.

4. Upstream keepalive — a module that wraps get/free

Wrapping: save the original pointer, install your own

The keepalive module isn't a balancing method. It wraps the get/free of whatever method is already in place (round robin, least_conn …) in one more layer: before get it hands out a cached connection, and after free it puts the connection into the cache. It's the same trick as Part 10's filters, which saved the previous filter in ngx_http_next_body_filter and put themselves on top.

The wrapping happens twice: once after the config has been read (wrapping peer.init) and once per request (wrapping get and free).

    for (i = 0; i < umcf->upstreams.nelts; i++) {

        /* skip implicit upstreams */
        if (uscfp[i]->srv_conf == NULL) {
            continue;
        }

        kcf = ngx_http_conf_upstream_srv_conf(uscfp[i],
                                            ngx_http_upstream_keepalive_module);

        if (kcf->max_cached == 0) {
            continue;
        }

        ngx_conf_init_msec_value(kcf->time, 3600000);
        ngx_conf_init_msec_value(kcf->timeout, 60000);
        ngx_conf_init_uint_value(kcf->requests, 1000);

        if (kcf->max_cached == NGX_CONF_UNSET_UINT) {
            kcf->local = 1;
            kcf->max_cached = 32;
        }

        kcf->original_init_peer = uscfp[i]->peer.init;

        uscfp[i]->peer.init = ngx_http_upstream_init_keepalive_peer;

src/http/modules/ngx_http_upstream_keepalive_module.c lines 534–559

This is the keepalive module's init_main_conf. ngx_http_block() calls each HTTP module's init_main_conf in module order (ngx_http.c lines 264–265), and in our build's objs/ngx_modules.c, ngx_http_upstream_module is on line 111 and ngx_http_upstream_keepalive_module on line 146. So keepalive runs after section 1's loop has set peer.init to round robin (or least_conn …), saves it in original_init_peer, and installs its own function. Module order decides wrapping order.

At request time, ngx_http_upstream_init_keepalive_peer() does the same thing to get/free.

    if (kcf->original_init_peer(r, us) != NGX_OK) {
        return NGX_ERROR;
    }

    kp->conf = kcf;
    kp->upstream = r->upstream;
    kp->data = r->upstream->peer.data;
    kp->original_get_peer = r->upstream->peer.get;
    kp->original_free_peer = r->upstream->peer.free;

    r->upstream->peer.data = kp;
    r->upstream->peer.get = ngx_http_upstream_get_keepalive_peer;
    r->upstream->peer.free = ngx_http_upstream_free_keepalive_peer;

src/http/modules/ngx_http_upstream_keepalive_module.c lines 174–186

Notice that peer.data is swapped too. Round robin's state (rrp) moves inside kp->data, and it is passed back out when calling the original, as in kp->original_get_peer(pc, kp->data). That way round robin never knows it has been wrapped.

Round robin always does the "picking a server"; keepalive only checks whether a connection to the chosen server already exists.

Example 2: seeing the wrapping in gdb

To confirm the calls really go that way, I set a breakpoint on ngx_http_upstream_get_round_robin_peer (ex/u12-gdb-wrap, run as a single process with master_process off; daemon off; so gdb can attach easily). It's a plain upstream {} with no keepalive line.

#0  ngx_http_upstream_get_round_robin_peer (pc=0x5555557afbd0, data=0x5555557b0100) at src/http/ngx_http_upstream_round_robin.c:699
#1  0x00005555556b0fac in ngx_http_upstream_get_keepalive_peer (pc=0x5555557afbd0, data=0x5555557b00c0) at src/http/modules/ngx_http_upstream_keepalive_module.c:219
#2  0x00005555555c003a in ngx_event_connect_peer (pc=0x5555557afbd0) at src/event/ngx_event_connect.c:34
#3  0x0000555555618faa in ngx_http_upstream_connect (r=0x55555577d750, u=0x5555557afbc0) at src/http/ngx_http_upstream.c:1606
#4  0x0000555555617771 in ngx_http_upstream_init_request (r=0x55555577d750) at src/http/ngx_http_upstream.c:868
#5  0x0000555555616a9f in ngx_http_upstream_init (r=0x55555577d750) at src/http/ngx_http_upstream.c:588
#6  0x00005555556099d6 in ngx_http_read_client_request_body (r=0x55555577d750, post_handler=0x555555616988 <ngx_http_upstream_init>) at src/http/ngx_http_request_body.c:84
#7  0x000055555567500c in ngx_http_proxy_handler (r=0x55555577d750) at src/http/modules/ngx_http_proxy_module.c:970
#8  0x00005555555ed617 in ngx_http_core_content_phase (r=0x55555577d750, ph=0x5555557a6a88) at src/http/ngx_http_core_module.c:1311
#9  0x00005555555ec448 in ngx_http_core_run_phases (r=0x55555577d750) at src/http/ngx_http_core_module.c:906
...
#15 0x00005555555cea51 in ngx_epoll_process_events (cycle=0x555555784120, timer=60000, flags=1) at src/event/modules/ngx_epoll_module.c:901
#16 0x00005555555bb0b9 in ngx_process_events_and_timers (cycle=0x555555784120) at src/event/ngx_event.c:248
#17 0x00005555555ca62b in ngx_single_process_cycle (cycle=0x555555784120) at src/os/unix/ngx_process_cycle.c:322
#18 0x000055555557f1b2 in main (argc=3, argv=0x7fffffffed38) at src/core/nginx.c:400

(Frames #10–#14 are request reading — from ngx_http_handler down to ngx_http_wait_request_handler — and are omitted. Addresses differ on every run.) Frame #1 is the point. Without a single keepalive line, ngx_http_upstream_get_keepalive_peer is wrapping round robin. And the data of #1 and #0 differ (...00c0 vs ...0100) — they are the kp and kp->data (= rrp) we just saw. Line 34 in #2 is the pc->get from section 1's table, and everything below it is exactly the content-phase path from Part 9 and the upstream start-up path from Part 11.

This version's default: keepalive is on even if you don't write it

In the init_main_conf above, when max_cached == NGX_CONF_UNSET_UINT — i.e. when no keepalive directive was written — it turns keepalive on with local = 1; max_cached = 32;. This change landed in March 2026.

$ git -C /d/openSource/nginx log --format='%h %ad %s' --date=short -3 -- src/http/modules/ngx_http_upstream_keepalive_module.c
d787755d5 2026-03-24 Upstream keepalive: fixed parameter parsing.
4fbe4b627 2026-03-23 Upstream: enabled keepalive by default for explicit upstreams.
c5d36eac3 2026-03-19 Upstream keepalive: distinguish cached connections by location.

git tag --contains 4fbe4b627 shows this commit first shipped in mainline 1.29.7 and has been in stable since 1.30.0 (I confirmed 1.30.6 has the same code). Talking HTTP/1.1 to backends by default (proxy_http_version 1.1, commit 6bb27a631, 2026-01-27 — ngx_http_proxy_merge_loc_conf() lines 4071–4072) also dates from 1.29.7. And the Connection header defaults to an empty value, so it isn't sent to the backend (lines 761–772; details in section 4 of Part 11).

So these three lines, which you see all the time in blog posts and older docs, give the same result in this version even if you leave them out:

# How upstream keepalive was enabled in older versions (1.29.6 and earlier) — the default in this version
upstream app { server 127.0.0.1:9001; keepalive 32; }
proxy_http_version 1.1;
proxy_set_header Connection "";

It's not exactly the same, though. When keepalive is enabled by default, local is attached. The commit message spells it out — "Cached keepalive connections are no longer shared between different locations referencing the same explicit upstream unless keepalive is explicitly configured without the "local" parameter." The tag in the next subsection is that distinction. To turn it off, write keepalive 0; (with max_cached == 0 it skips the wrapping with continue).

An implicit upstream — writing an address directly, as in proxy_pass http://127.0.0.1:9001;, with no upstream {} block — is caught by the loop's first check, srv_conf == NULL, and skipped ("skip implicit upstreams"). That's because ngx_http_upstream_add() creates implicit upstreams with ngx_pcalloc and never fills srv_conf (lines 6905–6915). Only ngx_http_upstream(), which reads an upstream {} block, does uscf->srv_conf = ctx->srv_conf; (line 6455).

The cache — a cache queue and a free queue

The 32 slots are created all at once at config time (lines 563–575); during requests they only move between the two queues. The queues are the ngx_queue_t from Part 3.

get first lets the original get pick a server, then looks for an idle connection to that address.

    /* ask balancer */

    rc = kp->original_get_peer(pc, kp->data);

    if (rc != NGX_OK) {
        return rc;
    }

    /* search cache for suitable connection */

    cache = &kp->conf->cache;

    for (q = ngx_queue_head(cache);
         q != ngx_queue_sentinel(cache);
         q = ngx_queue_next(q))
    {
        item = ngx_queue_data(q, ngx_http_upstream_keepalive_cache_t, queue);
        c = item->connection;

        if (kp->conf->local && item->tag != kp->upstream->conf) {
            continue;
        }

        if (ngx_memn2cmp((u_char *) &item->sockaddr, (u_char *) pc->sockaddr,
                         item->socklen, pc->socklen)
            == 0)
        {
            ngx_queue_remove(q);
            ngx_queue_insert_head(&kp->conf->free, q);

            goto found;
        }
    }

    return NGX_OK;

src/http/modules/ngx_http_upstream_keepalive_module.c lines 217–251

If it finds one, found: does pc->connection = c; pc->cached = 1; and returns NGX_DONE (lines 270–273). ngx_event_connect_peer() returns immediately on anything other than NGX_OK, so socket() and connect() are skipped, and ngx_http_upstream_connect() treats "rc == NGX_OK || rc == NGX_AGAIN || rc == NGX_DONE" (line 1650) the same way and uses u->peer.connection. If nothing is found it returns NGX_OK — meaning "make a new connection."

With local, the tag being compared is u->conf: the proxy settings (ngx_http_upstream_conf_t) of the location that handled the request. If two locations use the same upstream app, the connection cache is split in two.

Before caching a connection, free checks carefully whether it can be reused (lines 296–325). It doesn't cache it if the request failed, if the connection has already carried keepalive_requests (default 1000) requests, if it is older than keepalive_time (default 1 hour), or if the worker is shutting down. The most important check is u->keepalive. The protocol module (proxy) sets it when the response has been read to the end and the backend didn't say it would close the connection (u->keepalive = !u->headers_in.connection_close; — for example, line 2115). If there is no empty slot when caching, it closes the tail of the cache queue (the connection idle the longest) and reuses that slot (lines 334–350). An idle connection gets a keepalive_timeout (default 60 seconds) timer and its read handler becomes ngx_http_upstream_keepalive_close_handler, so it notices and cleans up if the backend closes first (lines 358–365). Whichever way it goes, at the end it calls round robin's free with kp->original_free_peer(pc, kp->data, state) (line 383) — section 3's fails and conns bookkeeping happens regardless of keepalive.

Easy to get wrong: if the backend answers in HTTP/1.0, nothing is reused

When I first built the example I tried to use python3's http.server as is for the backend. But python3's BaseHTTPRequestHandler answers in HTTP/1.0 unless told otherwise, and the proxy module treats a status line below HTTP/1.1 as "I'll close the connection."

    if (ctx->status.http_version < NGX_HTTP_VERSION_11) {

        if (ctx->status.code == NGX_HTTP_EARLY_HINTS) {
            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                          "upstream sent HTTP/1.0 response with early hints");
            return NGX_HTTP_UPSTREAM_INVALID_HEADER;
        }

        u->headers_in.connection_close = 1;
    }

src/http/modules/ngx_http_proxy_module.c lines 1874–1883

That leaves u->keepalive at 0, and free's !u->keepalive check rejects the connection. However correct the nginx config is, an HTTP/1.0 backend means no reuse. So I wrote the example backend to let me choose protocol_version, and case 4 below compares the two.

Example 3: confirming reuse from the client port the backend sees

Every time nginx connects to a backend, the OS assigns a new source port. So if the backend sees the same client port, nginx reused the same TCP connection. The backend prints that port in its response (ex/u12-keepalive).

# Example written for this post (not part of nginx's source)
worker_processes 1;
error_log /tmp/u12k/error.log debug;

http {
    # no keepalive line (on by default in this version: max_cached 32, local)
    upstream ka_default { server 127.0.0.1:9001; }

    # keepalive 0 — off
    upstream ka_off { server 127.0.0.1:9001; keepalive 0; }

    # the backend answers in HTTP/1.0
    upstream ka_http10 { server 127.0.0.1:9002; }

    # written without local — locations share the cache
    upstream ka_shared { server 127.0.0.1:9001; keepalive 8; }

    server {
        listen 8080;
        location /default/  { proxy_pass http://ka_default/; }
        location /default2/ { proxy_pass http://ka_default/; }
        location /off/      { proxy_pass http://ka_off/; }
        location /implicit/ { proxy_pass http://127.0.0.1:9001/; }
        location /http10/   { proxy_pass http://ka_http10/; }
        location /shared/   { proxy_pass http://ka_shared/; }
        location /shared2/  { proxy_pass http://ka_shared/; }
    }
}

(The real file writes the upstream blocks over several lines and also has a second explanatory comment line and pid, events {} and access_log off; lines.) 9001 answers in HTTP/1.1 and 9002 in HTTP/1.0. The result:

=== 1. explicit upstream, no keepalive line (default) ===
backend 9001 client-port 35724
backend 9001 client-port 35724
backend 9001 client-port 35724
=== 2. keepalive 0 ===
backend 9001 client-port 35738
backend 9001 client-port 35742
backend 9001 client-port 35744
=== 3. implicit upstream (proxy_pass http://127.0.0.1:9001/) ===
backend 9001 client-port 35746
backend 9001 client-port 35752
backend 9001 client-port 35756
=== 4. backend answers HTTP/1.0 ===
backend 9002 client-port 47662
backend 9002 client-port 47678
backend 9002 client-port 47686
=== 5. default (local): /default/ -> /default2/ -> /default/ ===
backend 9001 client-port 35724
backend 9001 client-port 35766
backend 9001 client-port 35724
=== 6. keepalive 8 (not local): /shared/ -> /shared2/ -> /shared/ ===
backend 9001 client-port 35774
backend 9001 client-port 35774
backend 9001 client-port 35774
=== debug log: keepalive lines ===
init keepalive peer:         12
using connection (reused):   6
saving connection (cached):  9

Port numbers differ on every run. Matching what we see to the source:

1. Default — all three requests on port 35724. The connection was reused in an upstream {} with no keepalive line.
2. keepalive 0 — a different port every time. With max_cached == 0, the continue skipped the wrapping.
3. Implicit upstream — same backend, yet a new connection every time. That's "skip implicit upstreams."
4. HTTP/1.0 backend — a new connection every time, even though keepalive is on by default. This is the "easy to get wrong" right above.
5. local — /default2/ got a new port (35766) even though it uses the same upstream, and going back to /default/ brought back case 1's 35724. The cache tag differs per location.
6. keepalive 8; (without local) — /shared2/ got the very connection /shared/ had used.

The debug-log counts add up too. The 12 init keepalive peer lines are the requests that went to wrapped upstreams (three each in groups 1, 4, 5 and 6); groups 2 and 3 have none. The 6 reuses are the 2nd and 3rd of group 1, the 1st and 3rd of group 5, and the 2nd and 3rd of group 6. The 9 cache insertions are one per response in groups 1, 5 and 6; group 4 (HTTP/1.0) never cached anything.

5. The stream module — the same foundation under TCP

A session is created right after accept

stream {} is a top-level block standing beside http {}, and its source lives separately in src/stream/. Yet even the file names pair up with HTTP — ngx_stream_handler.c (accepting connections), ngx_stream_core_module.c (phases), ngx_stream_proxy_module.c, ngx_stream_upstream_round_robin.c.

In Part 8, HTTP didn't create a request struct until data arrived, even after accepting a connection. Stream has no "request," so right after accept, ngx_stream_init_connection() creates a session (ngx_stream_session_t) from the connection pool immediately (s = ngx_pcalloc(c->pool, sizeof(ngx_stream_session_t)); — line 119), sets the read handler to ngx_stream_session_handler, and calls it right away (lines 180–205). The only thing that handler does is run the phase engine.

Seven phases and an engine of the same shape

typedef enum {
    NGX_STREAM_POST_ACCEPT_PHASE = 0,
    NGX_STREAM_PREACCESS_PHASE,
    NGX_STREAM_ACCESS_PHASE,
    NGX_STREAM_SSL_PHASE,
    NGX_STREAM_PREREAD_PHASE,
    NGX_STREAM_CONTENT_PHASE,
    NGX_STREAM_LOG_PHASE
} ngx_stream_phases;

src/stream/ngx_stream.h lines 84–92

That's fewer than HTTP's 11 (Part 9). With no URI there is no rewrite and no FIND_CONFIG to find a location. Instead there is PREREAD — the spot for peeking at the first bytes to decide where to send the connection (for example, ssl_preread reading the SNI from a TLS ClientHello). The engine has exactly the same shape as HTTP's:

void
ngx_stream_core_run_phases(ngx_stream_session_t *s)
{
    ngx_int_t                     rc;
    ngx_stream_phase_handler_t   *ph;
    ngx_stream_core_main_conf_t  *cmcf;

    cmcf = ngx_stream_get_module_main_conf(s, ngx_stream_core_module);

    ph = cmcf->phase_engine.handlers;

    while (ph[s->phase_handler].checker) {

        rc = ph[s->phase_handler].checker(s, &ph[s->phase_handler]);

        if (rc == NGX_OK) {
            return;
        }
    }
}

src/stream/ngx_stream_core_module.c lines 168–187

The handlers registered for each phase are flattened into a single array, and each slot's checker calls its handler and moves on depending on the result — line for line like Part 9's ngx_http_core_run_phases(). ngx_stream_init_phase_handlers(), which flattens the array (ngx_stream.c lines 330–385), gives the CONTENT phase exactly one slot and leaves the LOG phase out of the array — log handlers are called separately by ngx_stream_log_session() when the session ends (ngx_stream_handler.c lines 323–324).

That one CONTENT slot calls the server config's handler, and the proxy_pass directive fills it in — the same trick as HTTP's proxy_pass installing a location's content handler (Part 11).

    cscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_core_module);

    cscf->handler = ngx_stream_proxy_handler;

src/stream/ngx_stream_proxy_module.c lines 2766–2768

The proxy only moves bytes — but the upstream is the same

After finding the upstream, ngx_stream_proxy_handler() calls uscf->peer.init(s, uscf) exactly like HTTP and goes on to ngx_stream_proxy_connect() (lines 633–646). Inside, it calls the same ngx_event_connect_peer(&u->peer) (line 791). This is why section 1's ngx_peer_connection_t lives in src/event/. Stream's round robin (ngx_stream_upstream_round_robin.c) is a separate file, but the calculation is the same, and section 2's two lines are there verbatim (peer->current_weight += peer->effective_weight; on line 756, best->current_weight -= total; on line 781). Stream has no keepalive module, though — one client connection pairs with one backend connection to the end, so there is never a connection left over to reuse.

Once connected, ngx_stream_proxy_process(s, from_upstream, do_write) handles both directions with the same code. When from_upstream is 1, the source src is the backend connection (pc) and the destination dst is the client (c); when it's 0 the two are simply swapped (lines 1962–1985). After that it's a loop: read with src->recv() into a buffer, link it into an ngx_chain_t from Part 3, and send it out through ngx_stream_top_filter. It doesn't parse or rewrite headers the way HTTP does.

The two columns on top are separate; the foundation below is shared. This is the shape behind Part 2's "HTTP, stream and mail stand on the same core."

Example 4: a TCP proxy with stream

The backends are two python3 TCP servers (9101, 9102) that uppercase each line they receive and prefix it with their port. Not HTTP (ex/u12-stream).

# Example written for this post (not part of nginx's source)
stream {
    upstream echo {
        server 127.0.0.1:9101;
        server 127.0.0.1:9102;
    }

    # default: proxy_half_close off
    server {
        listen 7000;
        proxy_pass echo;
    }

    # wait for the full response even if the client closes only its write side
    server {
        listen 7001;
        proxy_pass echo;
        proxy_half_close on;
    }
}

(The real file also has a second explanatory comment line and worker_processes 1;, error_log, pid and events {} lines.) The result is below. The pid, ports, pointer addresses and connection numbers differ on every run.

=== 1. keep the client open for 1s, two sessions (port 7000) ===
9101: HELLO
9101: NGINX
9102: SECOND
=== 2. client closes its write side right after sending (port 7000, half_close off) ===
(nothing came back)
=== 3. same, but port 7001 with proxy_half_close on ===
9102: FOURTH
=== 4. debug log of the first session (selected lines) ===
[info] 23#23: *5 client 127.0.0.1:39326 connected to 0.0.0.0:7000
[debug] 23#23: *5 generic phase: 0
[debug] 23#23: *5 generic phase: 1
[debug] 23#23: *5 generic phase: 2
[debug] 23#23: *5 generic phase: 3
[debug] 23#23: *5 generic phase: 4
[debug] 23#23: *5 proxy connection handler
[debug] 23#23: *5 get rr peer, try: 2
[debug] 23#23: *5 get rr peer, current: 000056143A5BFB28 -1
[debug] 23#23: *5 connect to 127.0.0.1:9101, fd:11 #6
[debug] 23#23: *5 proxy connect: -2
[info] 23#23: *5 proxy 127.0.0.1:32856 connected to 127.0.0.1:9101
[info] 23#23: *5 client disconnected, bytes from/to client:12/24, bytes from/to upstream:24/12
[debug] 23#23: *5 finalize stream session: 200
=== 5. 'disconnected' lines of all sessions ===
[info] 23#23: *5 client disconnected, bytes from/to client:12/24, bytes from/to upstream:24/12
[info] 23#23: *7 client disconnected, bytes from/to client:7/13, bytes from/to upstream:13/7
[info] 23#23: *9 client disconnected, bytes from/to client:6/0, bytes from/to upstream:0/6
[info] 23#23: *11 upstream disconnected, bytes from/to client:7/13, bytes from/to upstream:13/7

In case 1 the two sessions alternated between 9101 and 9102 — section 2's round robin (with equal weights it simply alternates) runs in stream too. Reading case 4's debug log in order:

• client ... connected to — the ngx_log_error(NGX_LOG_INFO, ...) in ngx_stream_init_connection() (lines 149–151). The client hasn't sent anything yet.
• generic phase: 0 to 4 — the debug line of ngx_stream_core_generic_phase() (lines 201–202). The number isn't a phase number but the slot index in the flattened handler array (s->phase_handler). In our build, slots 0–4 hold five handlers from POST_ACCEPT (realip), PREACCESS (limit_conn, set), ACCESS (access) and SSL (ssl). ssl_preread, which would put a handler in PREREAD, isn't built, so PREREAD has no slot and slot 5 is CONTENT (ngx_stream_core_content_phase() prints no debug line and goes straight to proxy connection handler).
• get rr peer — the same string as HTTP. Evidence that stream's round-robin file is written to the same pattern as HTTP's.
• proxy connect: -2 — -2 is NGX_AGAIN: a non-blocking connect is in progress (just like the upstream connect in Part 11), and once it completes proxy ... connected to is printed.

Easy to get wrong: the session ends when the client closes only its write side

The first time I ran the example, printf 'hello\n' | nc -q1 127.0.0.1 7000 got nothing back. Connecting to the backend directly (nc -q1 127.0.0.1 9101) worked fine, but through nginx it came back empty. When stdin ends, nc -q1 closes only the write side of the socket (a half-close) and waits one second for an answer. nginx ended the session the moment it read EOF from the client.

    if (pc == NULL
| (!c->read->eof && !pc->read->eof)
| (!c->read->eof && c->buffered)
| (!pc->read->eof && pc->buffered))
    {
        return NGX_DECLINED;
    }

    if (pscf->half_close) {
        /* avoid closing live connections until both read ends get EOF */
        if (!(c->read->eof && pc->read->eof && !c->buffered && !pc->buffered)) {
             return NGX_DECLINED;
        }
    }

src/stream/ngx_stream_proxy_module.c lines 2199–2212

This is the TCP branch of ngx_stream_proxy_test_finalize(). The first if falls through to ending the session when "either side has hit EOF and nothing is left to send." Only proxy_half_close prevents that, and it defaults to off (ngx_conf_merge_value(conf->half_close, prev->half_close, 0); — line 2500). In case 5's output, *9 is that session — client disconnected, zero bytes sent to the client. *11 on 7001 (proxy_half_close on) waited until the backend sent its answer and closed, and ended with upstream disconnected. That's also why case 1 keeps stdin open for a second with (printf ...; sleep 1) | nc -q0. It's an easy trap when proxying, through stream, any protocol where the client "sends a request, closes its write side, then waits for the answer."

What I left out

• zone and run-time name resolution — sharing peers between workers in shared memory, and updating peers when DNS changes with server ... resolve. src/http/modules/ngx_http_upstream_zone_module.c (ngx_http_upstream_zone_copy_peers(), ngx_http_upstream_zone_resolve_handler()), and the NGX_HTTP_UPSTREAM_ZONE blocks in round robin.
• sticky — pinning a client to the same server with a cookie and so on. ngx_http_upstream_sticky_module.c (the NGX_HTTP_UPSTREAM_SID block we cut as /* ... */ is its slot). It wraps init_upstream, much like keepalive wraps init.
• hash ... consistent (ketama ring), random two, least_time — ngx_http_upstream_init_chash() and ngx_http_upstream_get_chash_peer() in ngx_http_upstream_hash_module.c, ngx_http_upstream_random_module.c, ngx_http_upstream_least_time_module.c.
• Falling over to backup servers — the failed: branch of ngx_http_upstream_get_round_robin_peer() (lines 772–806).
• TLS session reuse — the set_session/save_session pointers (ngx_http_upstream_set_round_robin_peer_session()). The keepalive module wraps these too.
• max_conns — seen only as the fourth filter in section 2. The peer struct's slow_start field is never set or read anywhere in this source (grep -rn slow_start src/ finds only four declaration lines).
• Stream over UDP (listen ... udp, proxy_responses), ssl_preread, ngx_stream_pass_module, and the mail module (src/mail/).

Summary

Wrapping up the series

With this post, all 12 parts on the map in Part 1 are written. Here is the whole series.

One request's path, once more

Let's retrace the main path drawn in Part 1, this time with part numbers. When you run nginx, main() reads the config and builds a cycle (Part 4); along the way each directive fills a module's config struct (Part 5), and all that memory comes from pools (Part 3). The master starts the workers and waits for signals (Part 6). Inside a worker, the event loop wakes from epoll_wait and accepts the connection with ngx_event_accept (Part 7); when data arrives it creates an ngx_http_request_t and reads the request line and headers (Part 8). ngx_http_core_run_phases walks the phases and picks a location (Part 9); a content handler sends a file, which passes through the filter chain and leaves via sendfile (Part 10), or, with proxy_pass, upstream connects to a backend and fetches the response (Part 11). Picking that backend, and reusing an already-open connection to it, was this post. When the response is done, the connection goes back to keep-alive to wait for the next request (Part 8) or is closed.

Three things kept coming back across these 12 parts. Tables of function pointers were nginx's consistent way of "plugging in someone else's code" — modules (Part 5), phase checkers (Part 9), the filter chain (Part 10), upstream callbacks (Part 11), and this post's get/free. Pools decided how long things live — the per-worker peer array in the config pool was one consequence. A single event loop forced every task to be cut into pieces that never wait; that's why a non-blocking connect returns NGX_AGAIN and is called back later. Knowing these three, you can guess where to start reading even an nginx module you've never seen.

Compared with the httpd and Tomcat series

This blog has already read Digging into Apache httpd's Source (16 parts) and Digging into Tomcat's Source (16 parts). The three servers solved the same problems differently.

• How connections are divided up. httpd swaps MPMs to choose a process/thread model (httpd Part 8), and Tomcat splits the work among Acceptor, Poller and a worker thread pool (Tomcat Part 5). nginx runs every connection on one event loop per worker (Part 7). Tomcat's Poller watching many connections with one selector resembles nginx, but Tomcat hands work to a worker thread when something happens, while nginx handles it right there.
• Processing stages. httpd's hooks (httpd Part 10) call the functions modules registered, stage by stage; nginx's phases (Part 9) flatten them into one array at config time. Tomcat puts a valve chain on each container (Tomcat Part 8).
• Response flow. httpd's bucket brigades (httpd Part 11) and nginx's ngx_chain_t filters (Part 10) share the idea of "passing a chain of buffers through filters."
• Load balancing. This post's smooth weighted round robin is the same calculation as byrequests in httpd Part 14 — httpd does lbstatus += lbfactor, picks the largest, and subtracts the total from the one picked. The difference is how methods are plugged in: httpd looks up an lbmethod provider by name, while nginx has a directive swap the init_upstream function pointer. For backend connection pools, httpd uses APR's apr_reslist per worker, and nginx uses this post's cache and free queues.

What to read next

This blog's open-source reading continues after Spring, Apache httpd, Tomcat and nginx with MariaDB. Having read four of the servers that take requests at the front, we now go inside the database server where those requests finally land.

• Digging into MariaDB's Source, Part 1: How I Read It, and the Map
• nginx Part 1: How I Read It with AI, and the Map
• nginx Part 2: The Big Picture — Master and Workers, Modules, the Cycle and Config Contexts
• httpd Part 14: mod_proxy — Reverse Proxy and Load Balancer