Digging into nginx's Source 11: Upstream and Proxy — One Request Through the Reverse Proxy

What this post covers

This is the eleventh post in the "Digging into nginx's Source" series. In Part 9 a request went through the 11 phases, picked its location, and reached the CONTENT phase, which calls the content handler. In Part 10 that handler served a static file, and the response went through the filter chain and out via sendfile. This time the content handler is proxy_pass: nginx doesn't produce the response itself, it sends the request again to another server behind it (the backend) and hands the backend's response back to the client. That is a reverse proxy.

On the map from Part 2, this post zooms into the upstream · proxy box in the "C. One HTTP request" row.

The blue box is this post. At Part 9's CONTENT phase we branch here instead of to static files (Part 10); "which server do we connect to" is Part 12's job.

We'll answer four questions.

• What does a single line proxy_pass http://127.0.0.1:9000; set up in the source — and how do the proxy module and ngx_http_upstream.c split the work?
• While the worker connects to the backend and waits for its answer, why don't other requests stall? In what order are functions called for one request?
• What does the request the backend receives actually look like — HTTP/1.1, Host, and the Connection header that is not sent?
• If the backend is down or slow, where are 502 and 504 decided and when does nginx retry another server? How does proxy_buffering change the flow of the response?

All line numbers in this post refer to this commit. How to get the same source and build the Docker image used for the examples is in Part 1.

ngx_http_upstream.c alone is 7,405 lines and ngx_http_proxy_module.c is 5,472. Many features live in the same files: caching, TLS to the backend, WebSocket upgrades. This post picks only the main path — send one GET request to an HTTP/1.x backend and bring the response back — and lists what was left out, with names and locations only, under "What this post leaves out" at the end.

Terms to know first

upstream is nginx's name for a server behind it, i.e. a backend. The upstream {} block in the config is a group of them, and in the source the word points at two things. One is the configuration side, the server group (ngx_http_upstream_srv_conf_t). The other is ngx_http_upstream_t, created once per request: a struct holding the whole state of "send this request to a backend and fetch the response". The second one is the star of this post.

Content handler is the function that produces the response body in the CONTENT phase. If the location config (ngx_http_core_loc_conf_t) has a function pointer in its handler field, the CONTENT phase calls only that and ignores the other handlers (Part 9).

Callback is a promise: "leave me a function and I'll call it at the right moment". In C it is a function-pointer field in a struct. ngx_http_upstream_t has function pointers like create_request and process_header, and the proxy module puts its own functions there. We saw modules hooking in through tables of function pointers in Part 5; if the C syntax feels unfamiliar, httpd Part 2 may help.

peer is one server address to connect to. The per-request ngx_peer_connection_t (u->peer) holds the chosen address, the connection, and two function pointers: "pick me a server" (get) and "I'm done with it" (free). Who picks and by what rule is Part 12. For this post it's enough to think "call get and an address comes out".

Non-blocking connect. If a socket is non-blocking and you call connect(), it doesn't wait for the connection to finish; it returns -1 right away with EINPROGRESS ("in progress"). When the connection completes, an event arrives saying the socket has become writable. In nginx, where one worker runs thousands of connections on one event loop (Part 7), backend connections have to work this way too.

event pipe is ngx_event_pipe_t in src/event/ngx_event_pipe.c: a "pipe" that reads from the backend socket and writes to the client socket. In between it keeps several buffers, plus a temporary file for when the buffers run out. It is used with proxy_buffering on (the default). Buffers (ngx_buf_t) and buffer chains (ngx_chain_t) were covered in Part 3.

1. What one proxy_pass line sets up

At config time: a content handler and an upstream

When the config parser meets the proxy_pass directive, it calls the proxy module's ngx_http_proxy_pass() (how a directive name leads to a function is in Part 5).

    if (plcf->upstream.upstream || plcf->proxy_lengths) {
        return "is duplicate";
    }

    clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);

    clcf->handler = ngx_http_proxy_handler;

    if (clcf->name.len && clcf->name.data[clcf->name.len - 1] == '/') {
        clcf->auto_redirect = 1;
    }

    /* ... */
    ngx_memzero(&u, sizeof(ngx_url_t));

    u.url.len = url->len - add;
    u.url.data = url->data + add;
    u.default_port = port;
    u.uri_part = 1;
    u.no_resolve = 1;

    plcf->upstream.upstream = ngx_http_upstream_add(cf, &u, 0);
    if (plcf->upstream.upstream == NULL) {
        return NGX_CONF_ERROR;
    }

src/http/modules/ngx_http_proxy_module.c lines 4391–4465

The function does three things.

1. It sets this location's content handler to ngx_http_proxy_handler (line 4397). Note where the handler goes: not into the proxy module's own config but into the core module's location config (clcf). When a request lands in this location, ngx_http_update_location_config() copies it with r->content_handler = clcf->handler; (src/http/ngx_http_core_module.c lines 1431–1433), and the CONTENT phase checker (the per-phase driver function that calls that phase's handlers, Part 9) calls it and passes the return value to ngx_http_finalize_request() (lines 1309–1313).
2. If the location name ends in /, it turns on auto_redirect (lines 4399–4401). That is where the behaviour "a request for /plain to location /plain/ gets a 301 to /plain/" comes from.
3. It strips http:// from the URL and creates or finds an upstream with ngx_http_upstream_add() (line 4462). Even if you write the address directly without an upstream {} block, nginx internally creates an upstream named 127.0.0.1 with a single server. If an upstream {} block with the same name exists, that one is returned (src/http/ngx_http_upstream.c lines 6859–6903). One created without a block is often called an implicit upstream. As Part 12 shows, the upstream keepalive that is on by default in this version does not apply to it.

If the URL contains a $ variable (proxy_pass http://$backend;), ngx_http_script_variables_count() on line 4407 returns non-zero; nginx then doesn't create an upstream in advance but compiles the URL into a script and evaluates it per request (lines 4409–4430, elided above). This post only looks at the variable-free case.

ngx_http_proxy_set_vars() extracts pieces of the URL for later. If the port is the default (80), the value to send as Host is 127.0.0.1, otherwise 127.0.0.1:9000; the URL's path part (the /v1/ in http://127.0.0.1:9000/v1/) goes into vars.uri (lines 5446–5471).

Easy to get wrong: a path after proxy_pass, or not

proxy_pass http://127.0.0.1:9000; and proxy_pass http://127.0.0.1:9000/v1/; differ only by a trailing path, yet the URI sent to the backend differs. The fork is in the request-line part of ngx_http_proxy_create_request() (lines 1393–1418).

• No path (ctx->vars.uri.len == 0): the URI the client sent is appended exactly as received, before any normalisation (r->unparsed_uri). unparsed_uri = 1 is decided while computing the length, at lines 1263–1265.
• With a path: the leading part as long as the location name (loc_len) is cut off, vars.uri is put in its place, and the rest plus ?args follow. With location /api/ + proxy_pass …/v1/, /api/users becomes /v1/users.

In a regex location or inside if, "cut off as many characters as the location name" makes no sense, so a proxy_pass with a path there is rejected while reading the config with "proxy_pass" cannot have URI part in location given by regular expression … (lines 4475–4491 of the same function).

Example 1: the request line the backend received

We started a python3 backend (port 9000) that echoes back the request line and headers it received, and sent requests to three locations. This example was run with nginx 1.31.9 built as-is from the reference commit (9a7439fd79) in a Docker container (Debian bookworm, --with-debug). The example folder is ex/u11-proxy-basics.

# An example written for this post (not part of nginx's source)
# ... (the settings before it and http { cut; only server {} is shown, de-indented)
server {
    listen 8080;

    # 1) proxy_pass without a URI part — pass the client's URI as is
    location /plain/ {
        proxy_pass http://127.0.0.1:9000;
    }

    # 2) proxy_pass with a URI part — replace the location prefix /api/ with /v1/
    location /api/ {
        proxy_pass http://127.0.0.1:9000/v1/;
    }

    # 3) add a header with proxy_set_header and change Host
    location /hdr/ {
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

=== 1. proxy_pass without URI part: /plain/a?x=1 ===
request line: GET /plain/a?x=1 HTTP/1.1
  Host: 127.0.0.1:9000
  User-Agent: curl/7.88.1
  Accept: */*
=== 2. proxy_pass with URI part: /api/users?id=7 ===
request line: GET /v1/users?id=7 HTTP/1.1
  Host: 127.0.0.1:9000
  User-Agent: curl/7.88.1
  Accept: */*

/plain/a?x=1 went through unchanged and /api/users?id=7 became /v1/users?id=7. In both cases the request line ends in HTTP/1.1 and Host is 127.0.0.1:9000. Where those two come from is section 4 (along with the result for location 3).

The difference a path after proxy_pass makes. The green boxes are the request lines from Example 1's output above (ngx_http_proxy_module.c lines 1393–1418).

2. How the work is split — ngx_http_upstream_t and callbacks

Besides proxy_pass, fastcgi_pass, uwsgi_pass, scgi_pass, grpc_pass and memcached_pass all do the same job: "connect to a backend, send the request, fetch the response". What differs is only which bytes the request is made of and how the head of the response is read. nginx writes the shared part once in ngx_http_upstream.c and lets each module plug in only the differing part via function pointers.

The yellow box is the struct created once per request; the blue boxes are the functions the proxy module plugs in.

ngx_http_upstream_s — only the fields we need

struct ngx_http_upstream_s {
    ngx_http_upstream_handler_pt     read_event_handler;
    ngx_http_upstream_handler_pt     write_event_handler;

    ngx_peer_connection_t            peer;

    ngx_event_pipe_t                *pipe;

    ngx_chain_t                     *request_bufs;

    ngx_output_chain_ctx_t           output;
    ngx_chain_writer_ctx_t           writer;

    ngx_http_upstream_conf_t        *conf;
    ngx_http_upstream_srv_conf_t    *upstream;
    /* ... */
    ngx_int_t                      (*input_filter_init)(void *data);
    ngx_int_t                      (*input_filter)(void *data, ssize_t bytes);
    void                            *input_filter_ctx;

    /* ... */
    ngx_int_t                      (*create_request)(ngx_http_request_t *r);
    ngx_int_t                      (*reinit_request)(ngx_http_request_t *r);
    ngx_int_t                      (*process_header)(ngx_http_request_t *r);
    void                           (*abort_request)(ngx_http_request_t *r);
    void                           (*finalize_request)(ngx_http_request_t *r,
                                         ngx_int_t rc);
    /* ... */
    unsigned                         buffering:1;
    unsigned                         keepalive:1;
    unsigned                         upgrade:1;
    unsigned                         error:1;

    unsigned                         request_sent:1;
    unsigned                         request_body_sent:1;
    unsigned                         request_body_blocked:1;
    unsigned                         header_sent:1;
    unsigned                         response_received:1;
};

src/http/ngx_http_upstream.h lines 342–426

Read it in three groups.

• read_event_handler and write_event_handler at the top are the functions to call when an event arrives on the backend connection. As section 3 shows, they change from stage to stage. peer is the ngx_peer_connection_t mentioned above, pipe is the event pipe used when buffering, and request_bufs is the chain of request bytes to send to the backend.
• The function pointers in the middle are the callbacks a module fills. create_request builds the request bytes, process_header reads the head of the response, input_filter_init and input_filter are body filters that track the length and cut the body into pieces, and finalize_request cleans up at the end. reinit_request resets the module's state when retrying another server.
• The bit fields at the bottom are progress flags. Values like request_sent (have we started sending the request?) and header_sent (have we sent the response header to the client?) feed the retry decision in section 5. keepalive marks whether this connection may be reused; Part 12's keepalive module reads it.

Where the proxy module fills them — ngx_http_proxy_handler()

This is the content handler installed in section 1. It runs once per request.

    if (ngx_http_upstream_create(r) != NGX_OK) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
    /* ... */
    u->create_request = ngx_http_proxy_create_request;
    u->reinit_request = ngx_http_proxy_reinit_request;
    u->process_header = ngx_http_proxy_process_status_line;
    u->abort_request = ngx_http_proxy_abort_request;
    u->finalize_request = ngx_http_proxy_finalize_request;
    /* ... */
    u->buffering = plcf->upstream.buffering;

    u->pipe = ngx_pcalloc(r->pool, sizeof(ngx_event_pipe_t));
    if (u->pipe == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }

    u->pipe->input_filter = ngx_http_proxy_copy_filter;
    u->pipe->input_ctx = r;

    u->input_filter_init = ngx_http_proxy_input_filter_init;
    u->input_filter = ngx_http_proxy_non_buffered_copy_filter;
    u->input_filter_ctx = r;
    /* ... */
    rc = ngx_http_read_client_request_body(r, ngx_http_upstream_init);

    if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
        return rc;
    }

    return NGX_DONE;

src/http/modules/ngx_http_proxy_module.c lines 894–976

1. ngx_http_upstream_create() allocates one ngx_http_upstream_t from the request pool and hangs it on r->upstream (lines 516–545). Coming from the request pool, it disappears together with the request.
2. It fills five callbacks (lines 931–935). process_header starts out as the status-line parser.
3. It allocates an event pipe and installs separate body filters for the buffered and the unbuffered case (lines 948–958).
4. Finally it calls ngx_http_read_client_request_body(r, ngx_http_upstream_init). As the name says, this means read the client's request body first, and when done, call the second argument, ngx_http_upstream_init. With no body, as with GET, it calls it right away. With the default proxy_request_buffering on, nginx receives the whole body into memory or a temp file before it starts connecting to the backend.

The handler returns NGX_DONE, meaning "not finished, but I'll finish it myself". The request isn't freed immediately because ngx_http_read_client_request_body() bumps the request's reference count r->main->count on its first line (src/http/ngx_http_request_body.c line 43). From here on the request is driven by events on the backend socket.

After the status line, process_header changes

ngx_http_proxy_process_status_line(), the first occupant of process_header, reads one line like HTTP/1.1 200 OK and then replaces itself with the header parser.

    if (ctx->status.http_version < NGX_HTTP_VERSION_11) {

        if (ctx->status.code == NGX_HTTP_EARLY_HINTS) {
            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                          "upstream sent HTTP/1.0 response with early hints");
            return NGX_HTTP_UPSTREAM_INVALID_HEADER;
        }

        u->headers_in.connection_close = 1;
    }

    u->process_header = ngx_http_proxy_process_header;

    return ngx_http_proxy_process_header(r);

src/http/modules/ngx_http_proxy_module.c lines 1874–1887

The upstream side only ever calls u->process_header(r) while reading the head of the response; whether it's the status line's turn or the headers' turn is remembered by the proxy module swapping the function pointer. It's the same trick as in Part 8, where reading the client's request switched the handler from ngx_http_process_request_line to ngx_http_process_request_headers. If the backend answers with HTTP/1.0, connection_close is set here so the connection won't be reused.

Modules using the same frame

Searching for files that call ngx_http_upstream_create(r) gives eight at this commit.

All but memcached — seven modules — start upstream with ngx_http_read_client_request_body(r, ngx_http_upstream_init) (memcached throws the request body away with ngx_http_discard_request_body() and calls ngx_http_upstream_init() directly). Lines 888–892 at the very top of ngx_http_proxy_handler() hand off to ngx_http_proxy_v2_handler() when proxy_http_version 2 is set (HTTP/2 backends exist in 1.30.x too). ngx_http_tunnel_module.c is a file that 1.30.6 doesn't have, and this post doesn't cover it.

Easy to get wrong: nobody calls abort_request

Every upstream module fills u->abort_request, but at this commit nothing in all of src/ ever calls that pointer (grep -rn 'abort_request' src finds only the struct declaration, the assignments and each module's function definition). The proxy one is an empty function that logs one debug line and returns (lines 2722–2728). What is actually called at the end is finalize_request, and in proxy that only logs too. Most of the cleanup is done on the upstream side (section 6).

3. One request's journey — init, connect, send, process_header

The dashed arrows are where the event loop wakes us up. In between, the worker runs other connections.

ngx_http_upstream_init_request() — build the request, get ready to pick a server

ngx_http_upstream_init() removes the read timer on the client connection and calls ngx_http_upstream_init_request() (lines 549–589). The lines from there that matter for this post:

    if (r->request_body) {
        u->request_bufs = r->request_body->bufs;
    }

    if (u->create_request(r) != NGX_OK) {
        ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
    }
    /* ... */
    if (uscf->peer.init(r, uscf) != NGX_OK) {
        ngx_http_upstream_finalize_request(r, u,
                                           NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
    }

    u->peer.start_time = ngx_current_msec;

    if (u->conf->next_upstream_tries
        && u->peer.tries > u->conf->next_upstream_tries)
    {
        u->peer.tries = u->conf->next_upstream_tries;
    }

    ngx_http_upstream_connect(r, u);

src/http/ngx_http_upstream.c lines 671–868

• The request bytes are built first, completely (line 675, section 4). Because they're built before connecting, a retry against another server sends the same buffer again.
• uscf->peer.init(r, uscf) prepares this request's server-picking state (line 854). Round robin is the default, and this is where u->peer.get and u->peer.free are filled in. The inside of it is Part 12. u->peer.tries (how many attempts are allowed, usually the number of servers) is set here too.
• Then comes ngx_http_upstream_connect().

The elided lines 679–853 contain proxy_bind, socket options, output chain setup, and the DNS lookup (resolver) for cases like proxy_pass http://$host where the address is decided per request.

ngx_event_connect_peer() — the non-blocking connect

This function lives in src/event/. It's not HTTP-only: the stream and mail modules connect to their backends with the same function.

    rc = pc->get(pc, pc->data);
    if (rc != NGX_OK) {
        return rc;
    }
    /* ... */
    s = ngx_socket(pc->sockaddr->sa_family, type, 0);
    /* ... */
    c = ngx_get_connection(s, pc->log);
    /* ... */
    if (ngx_nonblocking(s) == -1) {
        ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno,
                      ngx_nonblocking_n " failed");

        goto failed;
    }

src/event/ngx_event_connect.c lines 34–105

First it picks a server with pc->get() (line 34), creates a socket and puts it into one of the free connection structs prepared in advance, worker_connections of them (Part 7), then makes it non-blocking. Backend connections come from the same connection array as client connections, so they share worker_connections. A worker used as a proxy spends two connections per request.

    rc = connect(s, pc->sockaddr, pc->socklen);

    if (rc == -1) {
        err = ngx_socket_errno;


        /* ... */

    if (ngx_add_conn) {
        if (rc == -1) {

            /* NGX_EINPROGRESS */

            return NGX_AGAIN;
        }

        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, pc->log, 0, "connected");

        wev->ready = 1;

        return NGX_OK;
    }

src/event/ngx_event_connect.c lines 221–277

ngx_add_conn is the function that registers a connection's read and write sides with epoll at once, edge-triggered (the epoll module fills this pointer in; the registration itself already happened at lines 211–215, before connect()). After that, connect()'s result splits three ways.

If pc->get() returns anything other than NGX_OK, that value is returned as is, before a socket is even created (lines 35–37). With no live server it's NGX_BUSY; when Part 12's keepalive module hands out a cached connection, it's NGX_DONE.

Easy to get wrong: even a closed port on the same machine starts "in progress"

Reading only the third row of the table, you'd expect connect() to 127.0.0.1:9000, where nobody listens, to return ECONNREFUSED immediately. Sending a request with Example 1's backend stopped and looking at the debug log says otherwise (step 7 of ex/u11-proxy-basics).

=== 7. backend stopped: where is 'Connection refused' noticed? ===
502
[debug] 157#157: *1 connect to 127.0.0.1:9000, fd:10 #2
[debug] 157#157: *1 http upstream connect: -2
[error] 157#157: *1 connect() failed (111: Connection refused) while connecting to upstream
[debug] 157#157: *1 http next upstream, 2

It started with connect: -2, i.e. EINPROGRESS, and the refusal was only discovered after the event loop woke us up again (the pid and connection numbers depend on the environment). What notices the refusal then is getsockopt(SO_ERROR) in ngx_http_upstream_test_connect() (lines 3073–3083). It's called before sending the request (lines 2185–2188) or before reading the response (lines 2485–2488); on error it logs the same text connect() failed and calls section 5's ngx_http_upstream_next() with NGX_HTTP_UPSTREAM_FT_ERROR (value 2 — the log's http next upstream, 2). Since the same text is logged from two places, tell them apart by the http upstream connect: value on the line just before.

ngx_http_upstream_connect() — install handlers and return

    c = u->peer.connection;

    c->requests++;

    c->data = r;

    c->write->handler = ngx_http_upstream_handler;
    c->read->handler = ngx_http_upstream_handler;

    u->write_event_handler = ngx_http_upstream_send_request_handler;
    u->read_event_handler = ngx_http_upstream_process_header;
    /* ... */
    if (rc == NGX_AGAIN) {
        ngx_add_timer(c->write, u->conf->connect_timeout);
        return;
    }
    /* ... */
    ngx_http_upstream_send_request(r, u, 1);

src/http/ngx_http_upstream.c lines 1652–1750

It sets the read and write event handlers of the backend connection both to ngx_http_upstream_handler (lines 1658–1659), and writes the real work separately into u->write_event_handler and u->read_event_handler (lines 1661–1662). On NGX_AGAIN it sets a connect_timeout timer on the write event and simply returns (lines 1736–1739). The worker goes back to the event loop and runs other connections; when the connection completes, epoll reports a write event.

ngx_http_upstream_handler() is a switchboard that looks at the event's direction and calls one of the two.

    if (ev->write) {
        u->write_event_handler(r, u);

    } else {
        u->read_event_handler(r, u);
    }

src/http/ngx_http_upstream.c lines 1346–1351

The event handler is installed once; when the stage changes, only u->…_event_handler changes. Those changes are the state machine of this whole post.

All names in the boxes are function names with ngx_http_upstream_ removed. r->write is the client side's r->write_event_handler.

Send, then set the read timer

When the write event arrives we go ngx_http_upstream_send_request_handler() → ngx_http_upstream_send_request(). On the first send, ngx_http_upstream_test_connect() checks whether the connect really succeeded (SO_ERROR) (lines 2185–2188), and the request chain is written. Once everything is written it ends like this.

    if (!u->request_body_sent) {
        u->request_body_sent = 1;

        if (u->header_sent) {
            return;
        }

        if (u->conf->ignore_input) {
            ngx_http_upstream_process_header(r, u);
            return;
        }

        ngx_add_timer(c->read, u->conf->read_timeout);

        if (c->read->ready) {
            ngx_http_upstream_process_header(r, u);
            return;
        }
    }

src/http/ngx_http_upstream.c lines 2265–2283

From now on it's reading's turn, so a proxy_read_timeout timer is set (line 2277). If not everything could be sent (NGX_AGAIN), a proxy_send_timeout timer is set and we wait for the next write event (lines 2204–2235). All three timeouts default to 60 seconds in ngx_http_proxy_merge_loc_conf() (lines 3731–3738).

Reading the head of the response — ngx_http_upstream_process_header()

    for ( ;; ) {

        n = c->recv(c, u->buffer.last, u->buffer.end - u->buffer.last);

        /* ... */

        if (n == 0) {
            ngx_log_error(NGX_LOG_ERR, c->log, 0,
                          "upstream prematurely closed connection");
        }

        if (n == NGX_ERROR || n == 0) {
            ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR);
            return;
        }

        /* ... */
again:

        rc = u->process_header(r);

        if (rc == NGX_AGAIN) {

            if (u->buffer.last == u->buffer.end) {
                ngx_log_error(NGX_LOG_ERR, c->log, 0,
                              "upstream sent too big header");

                ngx_http_upstream_next(r, u,
                                       NGX_HTTP_UPSTREAM_FT_INVALID_HEADER);
                return;
            }

            continue;
        }

src/http/ngx_http_upstream.c lines 2537–2602

On the first call it takes one buffer of proxy_buffer_size (default one page, usually 4KB) from the request pool (lines 2490–2501), reads as much as it can and passes it to u->process_header(r). If more is needed (NGX_AGAIN) it waits for the next read event, and if the buffer is full and the header still hasn't ended, it fails with upstream sent too big header. In other words, the whole response header must fit into a single proxy_buffer_size buffer.

Once the header is read, if the status is 300 or higher it first considers a retry (ngx_http_upstream_test_next()) or proxy_intercept_errors (lines 2632–2641); then ngx_http_upstream_process_headers() copies the backend's headers into r->headers_out for the client, and we move on to ngx_http_upstream_send_response() (section 6).

Example 2: the order seen through the debug log and gdb

From the same ex/u11-proxy-basics, here are only the upstream lines of the debug log for one /plain/one request (timestamp and pid at the start removed).

*11 http init upstream, client timer: 0
*11 http proxy header: "User-Agent: curl/7.88.1"
*11 http proxy header: "Accept: */*"
*11 http proxy header:
*11 connect to 127.0.0.1:9000, fd:10 #12
*11 http upstream connect: -2
*11 http upstream request: "/plain/one?"
*11 http upstream send request handler
*11 http upstream send request
*11 http upstream send request body
*11 http upstream request: "/plain/one?"
*11 http upstream process header
*11 http proxy status 200 "200 OK"
*11 http proxy header: "Server: BaseHTTP/0.6 Python/3.11.2"
*11 http proxy header: "Date: Sat, 07 Nov 2026 03:05:06 GMT"
*11 http proxy header: "Content-Type: text/plain"
*11 http proxy header: "Content-Length: 101"
*11 http proxy header: "X-Backend: echo"
*11 http proxy header done
*11 http upstream exit: 0000000000000000
*11 finalize http upstream request: 0
*11 close http upstream connection: 10

The date differs per run; the connection numbers (*11, #12) and the fd depend on how many connections came before and which files are open (re-running this script gave the same values). The functions above show up in order.

• http init upstream is the debug log in ngx_http_upstream_init() (lines 555–556); the http proxy header: lines were logged by section 4's ngx_http_proxy_create_request() while building the request.
• connect to 127.0.0.1:9000, fd:10 #12 is lines 218–219 of ngx_event_connect_peer(), and http upstream connect: -2 is line 1608. -2 is NGX_AGAIN, i.e. EINPROGRESS. The client connection is *11 and the newly opened backend connection #12 — both share one numbering counter.
• The next line, http upstream request:, is line 1338 of ngx_http_upstream_handler(). This is where the event loop called us back with a write event. Then send request handler → send request.
• The second http upstream request: is the callback with a read event, followed by process header → http proxy status 200 → header lines → header done.
• The 101-byte body arrived together with the header in the first read, so the event pipe had nothing more to read and ended with http upstream exit; being an implicit upstream, the connection was closed rather than cached (close http upstream connection: 10).

We ran the same request in a single process with master_process off; daemon off; and used gdb to stop at three functions and print the call stack (gdb starts the program with address randomization (ASLR) off, so the addresses came out the same on a re-run, but they change with a different build; argument lists were removed, and frames #8–#11 (from ngx_http_handler to ngx_http_process_request_line) were shortened to ...).

Breakpoint 2, ngx_event_connect_peer at src/event/ngx_event_connect.c:34
#0  ngx_event_connect_peer at src/event/ngx_event_connect.c:34
#1  0x0000555555618faa in ngx_http_upstream_connect at src/http/ngx_http_upstream.c:1606
#2  0x0000555555617771 in ngx_http_upstream_init_request at src/http/ngx_http_upstream.c:868
#3  0x0000555555616a9f in ngx_http_upstream_init at src/http/ngx_http_upstream.c:588
#4  0x00005555556099d6 in ngx_http_read_client_request_body at src/http/ngx_http_request_body.c:84
#5  0x000055555567500c in ngx_http_proxy_handler at src/http/modules/ngx_http_proxy_module.c:970
#6  0x00005555555ed617 in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:1311
#7  0x00005555555ec448 in ngx_http_core_run_phases at src/http/ngx_http_core_module.c:906
...
#12 0x00005555555f9765 in ngx_http_wait_request_handler at src/http/ngx_http_request.c:534
#13 0x00005555555cea51 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:901
#14 0x00005555555bb0b9 in ngx_process_events_and_timers at src/event/ngx_event.c:248
#15 0x00005555555ca62b in ngx_single_process_cycle at src/os/unix/ngx_process_cycle.c:322
#16 0x000055555557f1b2 in main at src/core/nginx.c:400
Breakpoint 3, ngx_http_proxy_process_status_line at src/http/modules/ngx_http_proxy_module.c:1812
#0  ngx_http_proxy_process_status_line at src/http/modules/ngx_http_proxy_module.c:1812
#1  0x000055555561b4bc in ngx_http_upstream_process_header at src/http/ngx_http_upstream.c:2588
#2  0x00005555556188f7 in ngx_http_upstream_handler at src/http/ngx_http_upstream.c:1350
#3  0x00005555555cea51 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:901
#4  0x00005555555bb0b9 in ngx_process_events_and_timers at src/event/ngx_event.c:248
#5  0x00005555555ca62b in ngx_single_process_cycle at src/os/unix/ngx_process_cycle.c:322
#6  0x000055555557f1b2 in main at src/core/nginx.c:400

(The stack for the first breakpoint, ngx_http_proxy_create_request, is left out: below #1 ngx_http_upstream_init_request … :675 it's the same as #2 onwards of the connect stack above.) Side by side, the two stacks show this section's point.

• Up to the connect, we're still inside the very call that read the client's request. ngx_http_wait_request_handler (Part 8) → the phase engine (Part 9) → ngx_http_proxy_handler (line 970) → … → ngx_event_connect_peer. A single event — the one that read the request — carries us all the way to starting the connection.
• The stack that reads the response is short. ngx_epoll_process_events calls ngx_http_upstream_handler (line 1350, the read side) directly. The call that started the connect has long since returned, and the worker could do other work in between. The request's state lives not on the stack but in the r->upstream struct.

4. The request sent to the backend — ngx_http_proxy_create_request()

Two passes: count the length, then copy

ngx_http_proxy_create_request() is over 400 lines, but its shape is simple. It walks the same ingredients twice. The first time it only adds up lengths (lines 1251–1370); with that length it allocates a buffer exactly once (ngx_create_temp_buf(r->pool, len), line 1373), then walks again in the same order and copies (lines 1386–1543). There's no reallocating as the buffer grows, and since the allocation comes from the request pool there's nothing to free separately either.

Yellow boxes are memory. The body data isn't copied; for each already-read buffer only a copy of its ngx_buf_t struct and a new chain link are created and appended (lines 1576–1598).

The default header table — what gets removed, what gets replaced

The "default table" among the ingredients is this one.

static char  ngx_http_proxy_version[] = " HTTP/1.0" CRLF;
static char  ngx_http_proxy_version_11[] = " HTTP/1.1" CRLF;


static ngx_keyval_t  ngx_http_proxy_headers[] = {
    { ngx_string("Host"), ngx_string("") },
    { ngx_string("Connection"), ngx_string("") },
    { ngx_string("Proxy-Connection"), ngx_string("") },
    { ngx_string("Content-Length"), ngx_string("$proxy_internal_body_length") },
    { ngx_string("Transfer-Encoding"), ngx_string("$proxy_internal_chunked") },
    { ngx_string("TE"), ngx_string("") },
    { ngx_string("Keep-Alive"), ngx_string("") },
    { ngx_string("Expect"), ngx_string("") },
    { ngx_string("Upgrade"), ngx_string("") },
    { ngx_null_string, ngx_null_string }
};

src/http/modules/ngx_http_proxy_module.c lines 757–772

The values from proxy_set_header are merged with this table (on the same name, the config wins: ngx_http_proxy_init_headers() lines 4266–4287). Client headers whose name is in this table are not copied over (the ngx_hash_find → continue at lines 1361–1365). And if the value is an empty string, the header isn't written at all (lines 1335–1337). So:

• Connection, Keep-Alive, Proxy-Connection, TE, Expect, Upgrade — removed even if the client sent them. These are hop-by-hop headers that belong only to the connection between client and nginx, and must not be passed on to the backend connection.
• Content-Length, Transfer-Encoding — replaced with values nginx recomputes ($proxy_internal_body_length etc.).
• Host — empty in the table but handled separately. As in lines 1239–1243 of the excerpt below, without proxy_set_header Host the host_header from section 1, i.e. $proxy_host, is used.

The version in the request line is decided in the same function.

    if (plcf->host_value == NULL
| (host.len == 0 && plcf->http_version == NGX_HTTP_VERSION_11))
    {
        host = ctx->vars.host_header;
    }
    /* ... */
    if (plcf->http_version == NGX_HTTP_VERSION_11) {
        b->last = ngx_cpymem(b->last, ngx_http_proxy_version_11,
                             sizeof(ngx_http_proxy_version_11) - 1);

    } else {
        b->last = ngx_cpymem(b->last, ngx_http_proxy_version,
                             sizeof(ngx_http_proxy_version) - 1);
    }

src/http/modules/ngx_http_proxy_module.c lines 1239–1429

Defaults that changed in this version — HTTP/1.1, and no Connection

proxy_http_version defaults to NGX_HTTP_VERSION_11 in ngx_http_proxy_merge_loc_conf() (lines 4071–4072). For a long time nginx sent HTTP/1.0 and Connection: close to backends, which is why the advice "to use upstream keepalive you must write proxy_http_version 1.1; and proxy_set_header Connection "";" is so widespread. In this version both are the default. Following it with git log -S, it changed like this.

In stable 1.30.6 the default table has Host as $proxy_internal_host and Connection as an empty value. So HTTP/1.1 and not sending Connection are in 1.30.x too; only the separate handling of Host is 1.31.x.

Easy to get wrong: switching to HTTP/1.0 doesn't send Connection: close either

With proxy_http_version 1.0; the request line becomes " HTTP/1.0" CRLF from lines 1427–1428, but the default table's Connection is still empty, so no Connection header is added (6bb27a631 removed the $proxy_internal_connection variable entirely). Since HTTP/1.0 closes the connection by default when the header is absent, the result is the same. We checked this only in the source and didn't run it as an example.

Example 1, continued: removed headers, changed Host, hidden Server

The client sent Host: shop.example, Connection: keep-alive and Keep-Alive: timeout=5 to location 3.

=== 3. proxy_set_header Host $host + X-Real-IP, client sends Connection: keep-alive ===
request line: GET /hdr/ HTTP/1.1
  Host: shop.example
  X-Real-IP: 127.0.0.1
  User-Agent: curl/7.88.1
  Accept: */*
=== 4. response headers seen by the client ===
HTTP/1.1 200 OK
Server: nginx/1.31.9
Date: Sat, 07 Nov 2026 03:05:06 GMT
Content-Type: text/plain
Content-Length: 98
Connection: keep-alive
X-Backend: echo

The Date value differs per run.

• The backend received neither Connection nor Keep-Alive. Host is the value replaced by proxy_set_header Host $host, and X-Real-IP was added. Headers from the config coming before the client's headers also follows the copy order above (config values at lines 1447–1499 → client headers at lines 1504–1539).
• Step 4 is the response header the client received. Instead of the backend's Server: BaseHTTP/0.6 Python/3.11.2 (in Example 2's debug log) we see Server: nginx/1.31.9. The proxy module's hide list ngx_http_proxy_hide_headers[] contains Date, Server and X-Accel-* (lines 775–785), so ngx_http_upstream_process_headers() doesn't copy them (lines 3189–3193), and nginx's header filter (Part 10) fills those slots. The backend's X-Backend came through unchanged.
• The debug log also records the whole request that was built (http proxy header:%N"…", lines 1563–1565). For /plain/one it was GET /plain/one HTTP/1.1, Host: 127.0.0.1:9000, User-Agent, Accept, a blank line — exactly the buffer in the figure.

5. Failure and retry — ngx_http_upstream_next()

The code in section 3 was dotted with ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_…): when the connect was refused (lines 1645–1648), when the write or read timer fired (lines 2418–2421, 2480–2483), when the response header was invalid (lines 2617–2620). FT stands for failure type; each kind of failure is one bit (ngx_http_upstream.h lines 20–35). What you write as proxy_next_upstream error timeout http_502; is exactly the OR of these bits, and the default is error timeout (lines 3849–3853).

    switch (ft_type) {

    case NGX_HTTP_UPSTREAM_FT_TIMEOUT:
    case NGX_HTTP_UPSTREAM_FT_HTTP_504:
        status = NGX_HTTP_GATEWAY_TIME_OUT;
        break;

    /* ... */
    default:
        status = NGX_HTTP_BAD_GATEWAY;
    }

    /* ... */
    timeout = u->conf->next_upstream_timeout;

    if (u->request_sent
        && (r->method & (NGX_HTTP_POST|NGX_HTTP_LOCK|NGX_HTTP_PATCH)))
    {
        ft_type |= NGX_HTTP_UPSTREAM_FT_NON_IDEMPOTENT;
    }

    if (u->peer.tries == 0
| ((u->conf->next_upstream & ft_type) != ft_type)
| (u->request_sent && r->request_body_no_buffering)
| (timeout && ngx_current_msec - u->peer.start_time >= timeout))

src/http/ngx_http_upstream.c lines 4655–4710

The four questions spell out the single if at lines 4707–4710. The POST part of the third question works by lines 4701–4705 adding the NON_IDEMPOTENT bit to ft_type, so that the second question's bit check catches it.

The order matters.

1. First it counts a failure for that server (lines 4622–4643, u->peer.free(..., NGX_PEER_FAILED)). It counts whether or not it retries. Only 403 and 404 are not counted as failures (NGX_PEER_NEXT). How many failures take a server out for a while is round robin's max_fails and fail_timeout, covered in Part 12.
2. It decides the status code. A timeout is 504; the rest, such as a refusal or invalid header, is 502 (failures that came in as http_500, http_503, http_403, http_404 or http_429 keep that code — lines 4662–4686, elided in the excerpt above). The "502 Bad Gateway" and "504 Gateway Time-out" the client sees come from here.
3. It checks whether a retry is allowed. Servers left to try (tries), the failure kind listed in proxy_next_upstream, not a POST that was already sent (so a request the backend may have processed isn't sent twice), and within proxy_next_upstream_timeout — then it retries.
4. A retry closes the current connection and calls ngx_http_upstream_connect() again (lines 4746–4768). pc->get() picks the next server and the same request buffer built in section 4 is sent. If the request had already been sent on the old connection, ngx_http_upstream_reinit() rewinds the buffers and resets the module's state via u->reinit_request (lines 1697–1703).

Example 3: 502, 504, and the next server

ex/u11-proxy-errors runs a live backend on 9000 and a backend on 9002 that sleeps 3 seconds before answering, and leaves 9001 empty.

# An example written for this post (not part of nginx's source)
# ... (the settings before it and http { cut; only its contents are shown, de-indented)
log_format up '$request_uri -> $status  upstream_addr="$upstream_addr" '
              'upstream_status="$upstream_status"';
# ... (access_log cut)

# nobody listens on 9001; 9000 is alive
upstream pair {
    server 127.0.0.1:9001;
    server 127.0.0.1:9000;
}

upstream pair_nonext {
    server 127.0.0.1:9001;
    server 127.0.0.1:9000;
}

server {
    listen 8080;

    location /down/ {
        proxy_pass http://127.0.0.1:9001;     # backend down → 502
    }

    location /slow/ {
        proxy_pass http://127.0.0.1:9002;     # backend that sleeps 3 seconds
        proxy_read_timeout 1s;                # wait only 1 second → 504
    }

    location /next/ {
        proxy_pass http://pair;               # on failure, try the next server (default: error timeout)
    }

    location /nonext/ {
        proxy_pass http://pair_nonext;
        proxy_next_upstream off;              # never move on to the next server
    }
}

=== 1. backend down ===
/down/ -> 502 in 0.001147s
=== 2. slow backend, proxy_read_timeout 1s ===
/slow/ -> 504 in 1.006190s
=== 3. upstream with a dead server, proxy_next_upstream default ===
/next/a -> 200 in 0.015762s
/next/b -> 200 in 0.040978s
/next/c -> 200 in 0.012087s
=== 4. same, proxy_next_upstream off ===
/nonext/a -> 502 in 0.000660s
/nonext/b -> 200 in 0.035330s
/nonext/c -> 200 in 0.003075s
=== access.log ===
/down/ -> 502  upstream_addr="127.0.0.1:9001" upstream_status="502"
/slow/ -> 504  upstream_addr="127.0.0.1:9002" upstream_status="504"
/next/a -> 200  upstream_addr="127.0.0.1:9001, 127.0.0.1:9000" upstream_status="502, 200"
/next/b -> 200  upstream_addr="127.0.0.1:9000" upstream_status="200"
/next/c -> 200  upstream_addr="127.0.0.1:9000" upstream_status="200"
/nonext/a -> 502  upstream_addr="127.0.0.1:9001" upstream_status="502"
/nonext/b -> 200  upstream_addr="127.0.0.1:9000" upstream_status="200"
/nonext/c -> 200  upstream_addr="127.0.0.1:9000" upstream_status="200"

The timings and the pid in error.log (32#32) differ per run (the / line at the top of access.log, from the startup check, was removed). The matching lines from error.log (the client/request details at the end are shortened):

[error] 32#32: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, ... upstream: "http://127.0.0.1:9001/down/" ...
[error] 32#32: *4 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 127.0.0.1, ... upstream: "http://127.0.0.1:9002/slow/" ...
[error] 32#32: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, ... upstream: "http://127.0.0.1:9001/next/a" ...
[warn] 32#32: *6 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, ... upstream: "http://127.0.0.1:9001/next/a" ...
[error] 32#32: *11 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, ... upstream: "http://127.0.0.1:9001/nonext/a" ...
[warn] 32#32: *11 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, ... upstream: "http://127.0.0.1:9001/nonext/a" ...

• Step 1 (502): as seen in section 3, connect() failed (111: Connection refused) was logged by ngx_http_upstream_test_connect(), and the trailing "while connecting to upstream" is the log->action that function sets (line 3080). With only one server, tries drops to 0 and the answer is 502 right away.
• Step 2 (504): it ended after 1 second. The proxy_read_timeout timer fired and ngx_http_upstream_process_header() called next with FT_TIMEOUT (lines 2480–2483); "upstream timed out" is lines 4645–4648 of ngx_http_upstream_next(). "while reading response header from upstream" is the action from line 2478.
• Step 3: /next/a left two servers in upstream_addr and 502, 200 in upstream_status. It failed on 9001 (recorded as 502) and was retried on 9000 within the same request, successfully. $upstream_addr is comma-separated because each attempt makes ngx_http_upstream_connect() push one more slot onto the r->upstream_states array (line 1591). After that, upstream server temporarily disabled (round robin line 1068 — Part 12) took 9001 out for a while, so /next/b and /next/c went straight to 9000.
• Step 4: with proxy_next_upstream off, /nonext/a, which failed on 9001, ends with 502 without retrying. The next_upstream bits are empty, so the check at line 4708 catches it.

Easy to get wrong: the 60 seconds we waited while running the example

The first version of this example built the backend on python's HTTPServer (handles one connection at a time). Then step 4's /nonext/b ended with 504 after 60 seconds. The cause was this version's upstream keepalive default. pair, defined with an upstream {} block, has keepalive on by default (Part 12), so after /next/c finished nginx kept its connection to 9000 open in the cache; the python server, which serves one connection at a time, sat waiting for the next request on that connection and never accepted the new connection opened for pair_nonext. So the default 60-second proxy_read_timeout fired. Switching the backend to ThreadingHTTPServer gave the results above. It was a limit on the backend side, not nginx, but it showed what "nginx keeps backend connections open" can mean for a backend.

6. Streaming the response — proxy_buffering on and off

Once the header is read, it's ngx_http_upstream_send_response().

    rc = ngx_http_send_header(r);

    if (rc == NGX_ERROR || rc > NGX_OK || r->post_action) {
        ngx_http_upstream_finalize_request(r, u, rc);
        return;
    }

    u->header_sent = 1;
    /* ... */
    if (!u->buffering) {
        /* ... */
        u->read_event_handler = ngx_http_upstream_process_non_buffered_upstream;
        r->write_event_handler =
                             ngx_http_upstream_process_non_buffered_downstream;
        /* ... */
    u->read_event_handler = ngx_http_upstream_process_upstream;
    r->write_event_handler = ngx_http_upstream_process_downstream;

    ngx_http_upstream_process_upstream(r, u);

src/http/ngx_http_upstream.c lines 3297–3622

The response header is handed to the header filter chain first, regardless of buffering (ngx_http_send_header() at line 3297). When it actually leaves through the socket is decided by Part 10's write filter. The fork after that is u->buffering, i.e. proxy_buffering (default on, lines 3707–3708).

• off: it keeps using the buffer the header was read into (a single proxy_buffer_size). Whatever is read goes through input_filter to ngx_http_output_filter(), and if the client can't take more, nothing more is read from the backend (ngx_http_upstream_process_non_buffered_request(), lines 3956–4085). With just one buffer, the backend is tied to the slow client's speed.
• on: it assembles an event pipe (lines 3503–3617, elided). The number and size of buffers come from proxy_buffers (default 8 × one page, lines 3753–3754); the temp file for when they run out goes under proxy_temp_path, at most proxy_max_temp_file_size. The backend's read event is set to process_upstream, the client's write event to process_downstream, and both end up calling ngx_event_pipe().

One turn of the event pipe

ngx_event_pipe() repeats "write out, then read" until there's nothing left to do (src/event/ngx_event_pipe.c lines 29–58). The interesting part is the order in which it finds a buffer to read into.

            if (p->free_raw_bufs) {
                /* ... */
            } else if (p->allocated < p->bufs.num) {
                /* ... */
                b = ngx_create_temp_buf(p->pool, p->bufs.size);
                /* ... */
            } else if (!p->cacheable
                       && !p->downstream_error
                       && p->downstream->data == p->output_ctx
                       && p->downstream->write->ready
                       && !p->downstream->write->delayed)
            {
                /* ... */
                p->upstream_blocked = 1;
                /* ... */
                break;

            } else if (p->cacheable
| p->temp_file->offset < p->max_temp_file_size)
            {
                /* ... */
                rc = ngx_event_pipe_write_chain_to_temp_file(p);
                /* ... */
            } else {
                /* ... */
                ngx_log_debug0(NGX_LOG_DEBUG_EVENT, p->log, 0,
                               "no pipe bufs to read in");

                break;
            }

src/event/ngx_event_pipe.c lines 224–301

Yellow boxes are where memory or the file is used; the red box is where there's neither a buffer nor room in the temp file, so reading stops and waits (③ also stops reading, but in order to send first).

This is where buffering shows its meaning. Even if the client is slow, nginx keeps reading from the backend. When the memory buffers are used up (the limit at ②), it checks whether the client can take data (③); if not, it moves buffers into the temp file to free them and keeps reading (④). The backend can hand over its response quickly and move on. An application server's worker thread or process is far more expensive than an nginx connection, so nginx deals with the slow client on its behalf.

Also, a partially filled buffer is not sent to the client. The loop that distributes the bytes read into buffers (lines 336–360) calls p->input_filter() to put a buffer on the outgoing list (p->in) only when the buffer is full. A partly filled buffer goes back to free_raw_bufs to wait for the next read. The only exceptions are when the response length is known and the buffer covers all of the remaining length (lines 435–450), and when the backend closed the connection (lines 457–474). That's great for moving large responses in chunks, but a response that trickles out sits still until a buffer fills.

Example 4: a response that comes line by line, and the temp file

The backend in ex/u11-proxy-buffering answers /stream by first sending Content-Length: 12 and then writing three lines, one, two, end, one second apart. The client prints how many seconds after the request each line arrived. /small/ deliberately has tiny memory buffers (proxy_buffers 2 4k); there we request a 32MB response with a client that doesn't read for 2 seconds (slowclient.py).

# An example written for this post (not part of nginx's source)
# ... (the settings before it, http { and access_log off cut)
proxy_temp_path /tmp/u11b/proxy_temp;

# ... (server { listen 8080; cut; only the locations are shown, de-indented)
location /buf/ {
    proxy_pass http://127.0.0.1:9000;     # proxy_buffering defaults to on
}

location /nobuf/ {
    proxy_pass http://127.0.0.1:9000;
    proxy_buffering off;                  # pass data to the client as it arrives
}

location /small/ {
    proxy_pass http://127.0.0.1:9000;
    proxy_buffer_size 4k;                 # deliberately small memory buffers
    proxy_buffers 2 4k;
    proxy_busy_buffers_size 4k;
}

=== 1. proxy_buffering on (default): /buf/stream ===
  +2.0s  one
  +2.1s  two
  +2.1s  end
=== 2. proxy_buffering off: /nobuf/stream ===
  +0.0s  one
  +1.1s  two
  +2.1s  end
=== 3. 32MB through 2 x 4k buffers to a client that does not read for 2s ===
  client read 33554555 bytes (headers included) after sleeping 2s
[warn] 21#21: *6 an upstream response is buffered to a temporary file /tmp/u11b/proxy_temp/0000000001 while reading upstream, client: 127.0.0.1, server: , request: "GET /small/big HTTP/1.0", upstream: "http://127.0.0.1:9000/small/big", host: "x"
--- files left in the temp dir
0

The timings vary a little per run.

• Buffering on: all three lines arrived together after 2 seconds. The backend sent one at 0 seconds, but 4 bytes neither fill a 4KB buffer nor reach the remaining length (12 bytes), so it never made it onto p->in. Only at 2 seconds, when end arrived and the 12 bytes were complete, did it go out. This is why server-sent events (SSE) and long-polling responses are said to need proxy_buffering off.
• Buffering off: the lines arrived at 0, 1 and 2 seconds, just as the backend wrote them.
• Temp file: both memory buffers filled and the client wasn't reading, so we reached ④, logged the warning an upstream response is buffered to a temporary file … and wrote to the file. That warning text is what ngx_http_upstream_send_response() puts into p->temp_file->warn when not caching (lines 3541–3543). After 2 seconds the client received all 32MB without loss.

Easy to get wrong: you can't see the temp file in the directory

At the end of the example we counted the files under proxy_temp, and got 0. Not because they were deleted when the request ended — the file lost its name right after it was created. A temp file that isn't for caching (persistent) has its name removed from the directory (unlink) right after opening.

    fd = open((const char *) name, O_CREAT|O_EXCL|O_RDWR,
              access ? access : 0600);

    if (fd != -1 && !persistent) {
        (void) unlink((const char *) name);
    }

src/os/unix/ngx_files.c lines 283–288

On Unix you can keep reading and writing an open file through its fd after unlink(); the disk space is returned when the last fd closes. Even if the worker dies, no leftover files remain. On the other hand, you can't find what's filling the disk with ls; you'd have to look at the worker's open files with a tool like lsof (we didn't check this part with an example).

Finishing — ngx_http_upstream_finalize_request()

When the body is done (p->upstream_done, lines 4491–4496), upstream finishes.

    u->finalize_request(r, rc);

    if (u->peer.free && u->peer.sockaddr) {
        u->peer.free(&u->peer, u->peer.data, 0);
        u->peer.sockaddr = NULL;
        /* ... */
    if (u->peer.connection) {

        /* ... */
        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "close http upstream connection: %d",
                       u->peer.connection->fd);

        if (u->peer.connection->pool) {
            ngx_destroy_pool(u->peer.connection->pool);
        }

        ngx_close_connection(u->peer.connection);
    }

src/http/ngx_http_upstream.c lines 4821–4861

It calls the module's finalize_request, tells the server-picking side "done" through u->peer.free(), and closes the connection if one remains. That's why Example 2, an implicit upstream, ended with close http upstream connection. With an upstream that has keepalive on, free puts the connection in the cache and clears u->peer.connection, so nothing is closed here — that swap is Part 12's topic. Finally it signals the end of the response to the filter chain with ngx_http_send_special(r, NGX_HTTP_LAST) and ends the request with ngx_http_finalize_request() (lines 4940–4955).

What this post leaves out

• Response caching (proxy_cache): ngx_http_upstream_cache() from ngx_http_upstream.c line 875, src/http/ngx_http_file_cache.c
• TLS to the backend (proxy_pass https://): ngx_http_upstream_ssl_init_connection() from line 1757
• Upgrade such as WebSocket (two-way relaying after a 101 response): ngx_http_upstream_upgrade() line 3627, ngx_http_upstream_process_upgraded() line 3713
• Reading the request body and proxy_request_buffering off: src/http/ngx_http_request_body.c, ngx_http_upstream_send_request_body() line 2288
• Body filters for chunked responses: ngx_http_proxy_chunked_filter() ngx_http_proxy_module.c line 2209, ngx_http_proxy_non_buffered_chunked_filter() line 2471
• X-Accel-Redirect (the backend making nginx do an internal redirect): ngx_http_upstream_process_headers() lines 3106–3168
• 103 Early Hints: ngx_http_upstream_process_early_hints() line 2663
• proxy_redirect and proxy_cookie_* (rewriting response headers): ngx_http_proxy_rewrite_redirect() line 2888, ngx_http_proxy_rewrite_cookie() line 2920
• proxy_pass http://$var, where the address is decided per request, and the resolver: ngx_http_proxy_eval() line 981, ngx_http_upstream_resolve_handler() ngx_http_upstream.c line 1244
• HTTP/2 backends: src/http/modules/ngx_http_proxy_v2_module.c

Summary

What's next

In this post we treated pc->get() as "call it and an address comes out", and put off the case where finalize caches the connection instead of closing it. The next post covers both. We'll see how smooth weighted round robin picks one of the servers in an upstream {} block, how max_fails and fail_timeout produce section 5's "temporarily disabled", and how the keepalive module wraps get/free to reuse connections. And we close the series with the stream {} module, which applies the same upstream idea to TCP. It continues in Part 12: Load Balancing, Upstream Keepalive and the Stream Module — and Wrapping Up the Series.

We saw the same job in httpd in httpd Part 14: mod_proxy. The idea of splitting a shared frame from protocol-specific parts is the same — in httpd, mod_proxy opens a scheme handler hook and mod_proxy_http queues up on it; in nginx, the proxy module plugs functions into the callback fields of ngx_http_upstream_t. The big difference is how they wait. In httpd's mod_proxy_http one worker thread holds the connection while it receives the backend's response and streams it to the client; nginx returns to the event loop at every connect and read, so one worker runs many proxied requests at once. The connection pool that reuses backend connections (httpd keeps an apr_reslist per worker) is, in nginx, Part 12's keepalive module. In Tomcat, the CoyoteAdapter of Tomcat Part 7, which passes a request from the connector to the container, is a handoff inside one process, so it sits in a different place from upstream, which sends the request across the network again.

• Part 9: The 11 Phases and Finding the location — In What Order a Request Is Handled
• Part 10: Filters and Static Files — How a Response Flows Out
• Part 7: The Event Loop — epoll, Timers, Posted Events and accept
• Part 3: Memory Pools and Core Data Structures — ngx_pool_t, ngx_str_t, ngx_buf_t and ngx_chain_t
• Apache httpd Part 14: mod_proxy