Digging into nginx's Source 3: Memory Pools and Core Data Structures — ngx_pool_t, ngx_str_t, ngx_buf_t and ngx_chain_t

What this post covers

This is the third post in the "Digging into nginx's Source" series. Part 2 drew the map of nginx as a whole: master and workers, modules, the cycle, and config contexts. From this post on we zoom into the boxes on that map one at a time. The first box is Memory pools · data structures, in the "A. Preparation" row of the Part 2 map.

The yellow box is this post. The memory and data structures every other box uses come from here.

There is a reason to open this box first. Almost every line of the nginx code we read from Part 4 on uses one of ngx_palloc(r->pool, …), ngx_str_t, ngx_array_push() or ngx_chain_t. Read that code without knowing them and you keep tripping over questions like "who gives this memory back, and when?" or "why doesn't it just print the string with printf("%s")?"

There are four questions to answer.

• Why does nginx use pools instead of malloc()/free(), and when is a pool born and when does it die?
• What does ngx_palloc() do inside — what happens when a block fills up, or when you ask for a big piece?
• How is nginx's string type ngx_str_t different from a C string, and why was it made that way?
• What do the array, list, queue and hash look like, and the ngx_buf_t·ngx_chain_t that carry responses?

Every line number in this post refers to that commit. Part 1 explains how to get the source at the same commit and the Docker setup used to run the examples. The pool and data-structure code alone (the .c and .h files of ngx_palloc, ngx_array, ngx_list, ngx_queue, ngx_hash, ngx_rbtree and ngx_buf) comes to 3,193 lines, and the string functions in ngx_string.[ch] add another 2,369. This post only picks the main thread you need to follow the code from Part 4 on.

Words to know first

A pool is a "bin" that lends out memory. Memory taken from a pool is not handed back piece by piece; it all goes back in one go when the pool is destroyed (ngx_destroy_pool()). nginx's type for it is ngx_pool_t. It is the same idea as the APR pool in httpd Part 4, but built quite differently (section 8 compares them).

A block is a chunk the pool gets in one go from a malloc-family function. The pool then hands out small pieces from inside the block. There is no separate "block" type in the nginx source; each block simply starts with (the front part of) an ngx_pool_t struct. This post calls them blocks for convenience.

A large allocation is a request too big to fit in a block. The pool doesn't put it in a block; it takes it separately with malloc() and records it on a list.

A cleanup is a function you leave with a pool, asking it to be called when the pool is destroyed. Tidying-up work that isn't "give memory back" — like closing a file you opened — goes here. If function pointers and void * are unfamiliar, httpd Part 2 explains them.

ngx_str_t is nginx's string. It has just two fields, a length (len) and a start address (data), and you must not expect a '\0' (NUL) at the end the way a C string has.

Buffers and chains — ngx_buf_t is a struct that records "where the data to send lives" (for memory, its start and end addresses; for a file, the file and a position), and ngx_chain_t is a singly linked list that strings those buffers together. A response travels out through the filters on such a chain.

Alignment means putting data at addresses the CPU likes to read from (say, multiples of 8). Put a pointer or an integer at an unaligned address and some CPUs slow down while others simply fault.

1. Why pools — nginx's memory dies with one of three lifetimes

Most of the data a web server creates while handling a request becomes useless the moment that request ends: the request struct, the header list, URI strings, per-request state for each module, the buffer for writing the response headers… Manage these one by one with malloc()/free() and dozens of modules all have to agree on "who calls free() and when". Miss a single spot and memory leaks on every request.

nginx changes the question. Instead of "when do I throw this piece away?", it has you choose "what does this piece die with?" There are three main lifetimes to choose from.

The cycle, from Part 2, is "the whole state of nginx running one set of configuration". The per-module config structs built from the config file, the list of listening sockets, the opened log files — all come from the cycle pool. So when nginx -s reload stands up a new cycle, throwing the old cycle's pool away wholesale makes the old configuration vanish cleanly. The details of reload come in Part 4 and Part 6.

The connection_pool_size and request_pool_size directives you may have seen in config files are exactly the sizes of the bottom two pools. Their defaults live in ngx_http_core_merge_srv_conf().

    ngx_conf_merge_size_value(conf->connection_pool_size,
                              prev->connection_pool_size, 64 * sizeof(void *));
    ngx_conf_merge_size_value(conf->request_pool_size,
                              prev->request_pool_size, 4096);

src/http/ngx_http_core_module.c lines 3622–3625

Both directives carry a checking function, ngx_http_core_pool_size(), which rejects the config if the value is smaller than NGX_MIN_POOL_SIZE (112 bytes on 64-bit) or not a multiple of 16 (lines 5504–5523).

The request pool is created in the same place as the request struct. The order is worth noticing.

    pool = ngx_create_pool(cscf->request_pool_size, c->log);
    if (pool == NULL) {
        return NULL;
    }

    r = ngx_pcalloc(pool, sizeof(ngx_http_request_t));
    if (r == NULL) {
        ngx_destroy_pool(pool);
        return NULL;
    }

    r->pool = pool;

src/http/ngx_http_request.c lines 584–595

The pool comes first, and the request struct r itself is taken from that pool. So destroying one pool at the end of the request makes r disappear too. The end of ngx_http_free_request(), which does the destroying, says so in a comment.

    /*
     * Setting r->pool to NULL will increase probability to catch double close
     * of request since the request object is allocated from its own pool.
     */

    pool = r->pool;
    r->pool = NULL;

    ngx_destroy_pool(pool);

src/http/ngx_http_request.c lines 4063–4071

Setting r->pool to NULL first is there to catch the bug of closing the same request twice. Near the top of the function, if r->pool == NULL it logs "http request already closed" and returns (lines 3994–3997). Since r itself lives in memory that has already been handed back, the comment honestly says it only "increases the probability".

A request pool lives shorter than a connection pool, and a connection pool shorter than the cycle pool. On one keep-alive connection, request pools come and go many times.

Easy to get wrong

• An nginx request pool is not a "child" of the connection pool. APR pools form a parent/child tree where destroying the parent destroys its children first (httpd Part 4), but ngx_pool_t has no field pointing to a parent (see the struct in section 2). The request pool and the connection pool are independent pools that know nothing of each other; the HTTP code destroys them separately, in the right order.
• 512 bytes for connection_pool_size is smaller than it sounds. As section 2 shows, the pool subtracts its own 80-byte header from that size, and any request bigger than the remaining 432 bytes doesn't go into a block but is taken separately with malloc(). The client_header_buffer_size buffer, 1KB by default, is exactly such a case. In the section 4 example you can see this in the debug log.

2. What ngx_pool_t looks like — a chain of blocks, a large list, a cleanup list

Here is the pool struct. Its name is ngx_pool_s, and ngx_pool_t is its typedef (nginx has a habit of defining struct ngx_xxx_s and calling it ngx_xxx_t).

typedef struct {
    u_char               *last;
    u_char               *end;
    ngx_pool_t           *next;
    ngx_uint_t            failed;
} ngx_pool_data_t;


struct ngx_pool_s {
    ngx_pool_data_t       d;
    size_t                max;
    ngx_pool_t           *current;
    ngx_chain_t          *chain;
    ngx_pool_large_t     *large;
    ngx_pool_cleanup_t   *cleanup;
    ngx_log_t            *log;
};

src/core/ngx_palloc.h lines 49–65

Read it in two layers.

• d (ngx_pool_data_t) — a header that every block has. last is the first byte not yet handed to anyone, end is the end of the block, next is the next block, and failed is the "number of times no room was found in this block" that section 3 covers.
• The fields after it — meaningful only once per pool. max is the biggest size a block can hand out, current is the block where allocation starts, chain collects used-up ngx_chain_t links (section 7), large is the list of big allocations, and cleanup is the list of tidy-up functions.

The first block has the whole ngx_pool_t as its header; from the second block on, only the d part at the front is used (ngx_palloc_block() in section 3). Both are pointed to as ngx_pool_t *, but in the second block allocations start right after d, so the max·current slots hold other data. Those fields must always be read from the first block only.

The link of the large list, ngx_pool_large_t, is a small struct with two fields, next and alloc; the link of the cleanup list, ngx_pool_cleanup_t, has three: a function pointer handler (void (*)(void *data)), data and next (ngx_palloc.h lines 30–46).

The function that creates a pool is short.

ngx_pool_t *
ngx_create_pool(size_t size, ngx_log_t *log)
{
    ngx_pool_t  *p;

    p = ngx_memalign(NGX_POOL_ALIGNMENT, size, log);
    if (p == NULL) {
        return NULL;
    }

    p->d.last = (u_char *) p + sizeof(ngx_pool_t);
    p->d.end = (u_char *) p + size;
    p->d.next = NULL;
    p->d.failed = 0;

    size = size - sizeof(ngx_pool_t);
    p->max = (size < NGX_MAX_ALLOC_FROM_POOL) ? size : NGX_MAX_ALLOC_FROM_POOL;

    p->current = p;
    p->chain = NULL;
    p->large = NULL;
    p->cleanup = NULL;
    p->log = log;

    return p;
}

src/core/ngx_palloc.c lines 18–43

1. It takes size bytes, 16-byte aligned, with ngx_memalign() (NGX_POOL_ALIGNMENT is 16, ngx_palloc.h line 24). On Linux this calls posix_memalign() and leaves posix_memalign: address:size @alignment in the debug log (src/os/unix/ngx_alloc.c lines 51–69).
2. It places the pool struct itself at the very front of that memory and sets last right after it. The pool struct is not malloced separately; it lives inside its own first block.
3. max is the smaller of "the block size minus the header" and NGX_MAX_ALLOC_FROM_POOL (ngx_pagesize - 1, usually 4095, ngx_palloc.h lines 16–20).

On 64-bit Linux sizeof(ngx_pool_t) is 80 bytes (we confirm this in the section 3 gdb example). That gives these max values for the default sizes.

Yellow is memory, green is the free space the next allocation will carve from, and blue is the tidy-up functions.

3. ngx_palloc — allocation is a pointer bump

There are four functions for taking memory from a pool: ngx_palloc() (aligned), ngx_pnalloc() (not aligned), ngx_pcalloc() (zero-filled) and ngx_pmemalign() (with an alignment of your choice, always large). ngx_pnalloc() is identical to the ngx_palloc() below except that the third argument it passes to ngx_palloc_small() is 0 instead of 1 (lines 135–145).

void *
ngx_palloc(ngx_pool_t *pool, size_t size)
{
#if !(NGX_DEBUG_PALLOC)
    if (size <= pool->max) {
        return ngx_palloc_small(pool, size, 1);
    }
#endif

    return ngx_palloc_large(pool, size);
}

src/core/ngx_palloc.c lines 122–132

If the size is at most max it comes from a block (ngx_palloc_small); otherwise it goes to malloc (ngx_palloc_large). The third argument, 1/0, says whether to align. Data read one byte at a time, like strings, needs no alignment, so ngx_pnalloc() saves a few bytes. Indeed ngx_pstrdup() uses ngx_pnalloc() (src/core/ngx_string.c lines 74–87).

#if !(NGX_DEBUG_PALLOC) means "only when pool debugging is off". With that macro on, every allocation goes to large and is malloced individually, so memory checkers such as Valgrind can see the boundaries of each piece. The configure scripts under auto/ don't mention this name (a search finds nothing), so to turn it on you pass it yourself as a compiler option, like --with-cc-opt=-DNGX_DEBUG_PALLOC=1. The build in this post does not.

The fast path: ngx_palloc_small

static ngx_inline void *
ngx_palloc_small(ngx_pool_t *pool, size_t size, ngx_uint_t align)
{
    u_char      *m;
    ngx_pool_t  *p;

    p = pool->current;

    do {
        m = p->d.last;

        if (align) {
            m = ngx_align_ptr(m, NGX_ALIGNMENT);
        }

        if ((size_t) (p->d.end - m) >= size) {
            p->d.last = m + size;

            return m;
        }

        p = p->d.next;

    } while (p);

    return ngx_palloc_block(pool, size);
}

src/core/ngx_palloc.c lines 148–174

1. Start at the current block.
2. If alignment is wanted, round last up to a multiple of NGX_ALIGNMENT (sizeof(uintptr_t), 8 on 64-bit, src/core/ngx_config.h lines 96–102). ngx_align_ptr is a bit-twiddling macro that "adds, then clears the low bits".
3. If the free space (end - m) is big enough, bump last by size and return the old position. That's it.
4. If not, try the next block. If none has room, make a new block.

Nearly every allocation ends at step 3: one comparison, one addition. Unlike malloc(), there is no searching for free space, no recording the piece's size for a later free(), no locking between threads. Since pieces are never handed back one by one, none of that is needed. The function is even marked ngx_inline to save the function-call cost. No locking also means one pool must not be shared by several threads — but an nginx worker runs a single event loop, so normally that isn't an issue (Part 7).

When a block is full: ngx_palloc_block and failed

static void *
ngx_palloc_block(ngx_pool_t *pool, size_t size)
{
    u_char      *m;
    size_t       psize;
    ngx_pool_t  *p, *new;

    psize = (size_t) (pool->d.end - (u_char *) pool);

    m = ngx_memalign(NGX_POOL_ALIGNMENT, psize, pool->log);
    if (m == NULL) {
        return NULL;
    }

    new = (ngx_pool_t *) m;

    new->d.end = m + psize;
    new->d.next = NULL;
    new->d.failed = 0;

    m += sizeof(ngx_pool_data_t);
    m = ngx_align_ptr(m, NGX_ALIGNMENT);
    new->d.last = m + size;

    for (p = pool->current; p->d.next; p = p->d.next) {
        if (p->d.failed++ > 4) {
            pool->current = p->d.next;
        }
    }

    p->d.next = new;

    return m;
}

src/core/ngx_palloc.c lines 177–210

1. Get a new block the same size as the first block (psize).
2. The new block uses only ngx_pool_data_t (32 bytes) as its header, and this allocation is carved right after it.
3. From current to the block before the last, bump failed by one. If the value before the bump was greater than 4 (it had already failed 5 times), move current to the following block.
4. Append the new block to the end of the chain.

What failed does is the point of this function. A block with a few dozen bytes left could still satisfy small allocations, so it is a shame to abandon it. But scanning nearly full blocks from the start every time would slow allocation down as blocks pile up. So nginx compromises: "the sixth time a new block had to be made while this block was in the way, treat it as full and skip it from now on." It gives up those leftover bytes, and in return allocation only ever looks at the last few blocks of the chain.

The green path is the one taken almost every time. Only the two yellow boxes call a malloc-family function.

Big allocations: ngx_palloc_large

    p = ngx_alloc(size, pool->log);
    /* ... */
    for (large = pool->large; large; large = large->next) {
        if (large->alloc == NULL) {
            large->alloc = p;
            return p;
        }

        if (n++ > 3) {
            break;
        }
    }

    large = ngx_palloc_small(pool, sizeof(ngx_pool_large_t), 1);
    /* ... */
    large->alloc = p;
    large->next = pool->large;
    pool->large = large;

    return p;
}

src/core/ngx_palloc.c lines 220–249

ngx_alloc() is a thin wrapper that calls malloc() and leaves malloc: address:size in the debug log (src/os/unix/ngx_alloc.c lines 17–31). The address is recorded on an ngx_pool_large_t link. If one of the first five slots on the list is empty (alloc == NULL, a slot emptied by ngx_pfree() from section 4), it is reused; otherwise a new link is taken as a small allocation from the pool and put at the front of the list. The big memory comes from malloc, while the 16-byte link that remembers it comes from the pool — the two paths are mixed.

Example 1: looking inside a request pool with gdb

This example was run with nginx 1.31.9 built straight from the base commit (9a7439fd79) in a Docker container (Debian bookworm, --with-debug, -O0 -g). To make gdb easy it was started as a single process with -g 'daemon off; master_process off;'; one static-file request was sent, execution stopped at the first line of ngx_http_static_handler(), and the gdb commands below were run. The full files are nginx.conf·pool.gdb·run.sh in the example folder ex/u03-pool-gdb/.

# Example written for this post (not part of the nginx source) — excerpt from pool.gdb
# (omitted: the set pagination/confirm lines, the echo lines that print the separators, the three prints of === 1., and the final kill/quit)
# walk the pool's block chain and print used bytes, free bytes and failed for each block
define poolwalk
  set $p = $arg0
  while $p
    printf "  block %p  used=%4ld  free=%4ld  failed=%lu\n", $p, (long)($p->d.last - (u_char *) $p), (long)($p->d.end - $p->d.last), $p->d.failed
    set $p = $p->d.next
  end
  printf "  current=%p  large=%p\n", $arg0->current, $arg0->large
end

break ngx_http_static_handler
run

print sizeof(ngx_pool_t)
print sizeof(ngx_http_request_t)
print *r->pool
poolwalk r->pool

# call ngx_palloc(r->pool, 3000) ourselves eight times
set $i = 0
while $i < 8
  call (void) ngx_palloc(r->pool, 3000)
  set $i = $i + 1
end
poolwalk r->pool

# 5000 bytes, bigger than max
print r->pool->max
set $big = ngx_palloc(r->pool, 5000)
print $big
print *r->pool->large

=== 2. the request pool ===
$4 = 80
$5 = 1360
$6 = {d = {last = 0x55555577e43c "ubject = \"/CN=openssl-cmp-test\"\nnewkey = insta.priv.pem\nout_trusted = apps/insta.ca.crt # does not include keyUsage digitalSignature\ncertout = insta.cert.pem\n\n[pbm] # Password-based protection for Ins"..., end = 0x55555577e700 "\020\020", next = 0x5555557aa320, failed = 0}, max = 4016, current = 0x55555577d700, chain = 0x0, large = 0x0, cleanup = 0x0, log = 0x555555780c20}
  block 0x55555577d700  used=3388  free= 708  failed=0
  block 0x5555557aa320  used=1152  free=2944  failed=0
  current=0x55555577d700  large=(nil)
=== 3. call ngx_palloc(r->pool, 3000) eight times ===
  block 0x55555577d700  used=3388  free= 708  failed=6
  block 0x5555557aa320  used=1152  free=2944  failed=6
  block 0x5555557ab330  used=3032  free=1064  failed=6
  block 0x5555557ac340  used=3032  free=1064  failed=5
  block 0x5555557ad350  used=3032  free=1064  failed=4
  block 0x5555557ae360  used=3032  free=1064  failed=3
  block 0x5555557af370  used=3032  free=1064  failed=2
  block 0x5555557b0380  used=3032  free=1064  failed=1
  block 0x5555557b1390  used=3032  free=1064  failed=0
  block 0x5555557b23a0  used=3032  free=1064  failed=0
  current=0x5555557ac340  large=(nil)
=== 4. call ngx_palloc(r->pool, 5000) -- larger than max ===
$7 = 4016
$8 = (void *) 0x5555557b33b0
$9 = {next = 0x0, alloc = 0x5555557b33b0}

(Addresses depend on the environment. gdb starts the program with address randomization (ASLR) turned off, so rerunning in the same container gave the very same addresses. The === 1. part is covered in section 5.)

There is a lot to read here.

• sizeof(ngx_pool_t) = 80, max = 4016. That matches section 2's arithmetic (4096 − 80).
• One request struct is 1,360 bytes. By the time we reach the static-file handler, 3,388 bytes of the first block are used and a second block already exists. Request struct, header list, the per-module ctx array, the variables array… the default 4KB request pool isn't enough in one block even for an ordinary GET request.
• Asking for 3,000 bytes eight times added eight blocks. A new block's used=3032 is a 32-byte header (ngx_pool_data_t) plus 3,000 bytes. You can see the header is 32 rather than 80 because it isn't the first block.
• failed and current. Every time a block is made, failed goes up by one from current to the block before the last, so earlier blocks have larger numbers. On the sixth call the first block dropped out of current, on the seventh the second, and on the eighth the third, leaving current at the fourth block (0x5555557ac340). The first block still has 708 bytes and the second 2,944 bytes free, yet small allocations no longer look there. The "compromise" above, in numbers.
• The 5,000 bytes exceeded max and went to large. The first link on the large list has an alloc equal to the address we got back.

Easy to get wrong

• Memory from ngx_palloc() is not empty. Look at d.last in the output above. The free space nobody has used yet contains stray text: "ubject = \"/CN=openssl-cmp-test\"…". It is what some earlier code (judging by the text, apparently whatever read an OpenSSL config file) wrote and abandoned in memory that posix_memalign() then returned. If you need zeroed memory, use ngx_pcalloc(). The request struct is taken with ngx_pcalloc() too (section 1).
• Sizes close to max waste blocks. Taking 3,000 bytes leaves 1,064 bytes in a 4KB block, and after a few failures that block gets skipped as well. If you often take big pieces close to 4KB from a request pool, raising request_pool_size is the better choice.

4. Giving back — destroy, cleanup, and ngx_pfree that only works for large

ngx_destroy_pool

void
ngx_destroy_pool(ngx_pool_t *pool)
{
    ngx_pool_t          *p, *n;
    ngx_pool_large_t    *l;
    ngx_pool_cleanup_t  *c;

    for (c = pool->cleanup; c; c = c->next) {
        if (c->handler) {
            ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0,
                           "run cleanup: %p", c);
            c->handler(c->data);
        }
    }

#if (NGX_DEBUG)
    /* ... debug-only loops that print "free: %p" and "free: %p, unused: %uz" ... */
#endif

    for (l = pool->large; l; l = l->next) {
        if (l->alloc) {
            ngx_free(l->alloc);
        }
    }

    for (p = pool, n = pool->d.next; /* void */; p = n, n = n->d.next) {
        ngx_free(p);

        if (n == NULL) {
            break;
        }
    }
}

src/core/ngx_palloc.c lines 46–96

1. Run the cleanups. It goes from the front of the list, and as we'll see below, cleanups are always added at the front, so in effect the last one registered runs first.
2. free() every large allocation.
3. free() every block. The pool struct lives in the first block, so the final ngx_free(p) gives back the pool itself. That is why it reads n = p->d.next before going round the loop.

The order in which a pool is destroyed. The log lines in the grey boxes come from the actual output of Example 2 below.

The #if (NGX_DEBUG) block (lines 61–81) is only compiled in with --with-debug; before actually freeing anything it logs free: address for each large allocation and free: address, unused: free bytes for each block. The comment explains why it doesn't log while freeing — "we could allocate the pool->log from this pool": the log struct itself might live in this pool.

The registering side:

    c = ngx_palloc(p, sizeof(ngx_pool_cleanup_t));
    /* ... */
    c->handler = NULL;
    c->next = p->cleanup;

    p->cleanup = c;

    ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, p->log, 0, "add cleanup: %p", c);

    return c;
}

src/core/ngx_palloc.c lines 316–339

It takes the link plus a size-byte data slot from the pool (the elided lines 321–329), and returns with handler still NULL. Filling in the function is the caller's job, and a link whose handler is NULL is skipped at destroy time. That enables a pattern of "reserve a slot first, fill in the handler only if the work succeeds". ngx_open_cached_file(), which opens static files, uses it exactly that way.

        cln = ngx_pool_cleanup_add(pool, sizeof(ngx_pool_cleanup_file_t));
        if (cln == NULL) {
            return NGX_ERROR;
        }

        rc = ngx_open_and_stat_file(name, of, pool->log);

        if (rc == NGX_OK && !of->is_dir) {
            cln->handler = ngx_pool_cleanup_file;
            clnf = cln->data;

            clnf->fd = of->fd;
            clnf->name = name->data;
            clnf->log = pool->log;
        }

src/core/ngx_open_file_cache.c lines 181–195

(This is the path when open_file_cache is off.) Only if the file opened successfully does it put ngx_pool_cleanup_file in handler. Here pool is the r->pool passed in by the static-file handler, so the file is closed when the request pool is destroyed. No code anywhere in request processing has to call close(). ngx_pool_cleanup_file() logs file cleanup: fd:N and calls ngx_close_file() (ngx_palloc.c lines 363–375).

ngx_pfree — only large allocations can be given back

ngx_int_t
ngx_pfree(ngx_pool_t *pool, void *p)
{
    ngx_pool_large_t  *l;

    for (l = pool->large; l; l = l->next) {
        if (p == l->alloc) {
            ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0,
                           "free: %p", l->alloc);
            ngx_free(l->alloc);
            l->alloc = NULL;

            return NGX_OK;
        }
    }

    return NGX_DECLINED;
}

src/core/ngx_palloc.c lines 277–294

This is a function APR doesn't have: it gives back a single piece. But only pieces on the large list work. Pass it a small piece taken from a block and it simply doesn't find it on the list, returns NGX_DECLINED, and does nothing. The emptied slot is left as alloc = NULL and reused by the next ngx_palloc_large() (section 3).

Keep-alive connections show why this function is needed. If thousands of keep-alive connections are sitting idle, having each of them hold a 1KB header buffer is a waste. ngx_http_set_keepalive() does this when a request finishes:

    /*
     * To keep a memory footprint as small as possible for an idle keepalive
     * connection we try to free c->buffer's memory if it was allocated outside
     * the c->pool.  The large header buffers are always allocated outside the
     * c->pool and are freed too.
     */

    b = c->buffer;

    if (ngx_pfree(c->pool, b->start) == NGX_OK) {

src/http/ngx_http_request.c lines 3463–3472

"If it was allocated outside c->pool (as large), give it back" ties into the "512-byte connection pool" point at the end of section 1. With the default config, the 1KB header buffer exceeds the connection pool's max (432), so it is always large and therefore can always be given back. Conversely, if you set a big connection_pool_size so the buffer fits in a block, this ngx_pfree() fails and the buffer stays until the connection is closed.

Example 2: watching pools come and go in the debug log

With nginx running one worker and error_log … debug;, curl sent two requests on one connection (a.html, b.html), and only the allocation and freeing lines were picked out of the debug log. The config and script are in the example folder ex/u03-pool-debug-log/. The environment is the same as Example 1, but this time nginx ran normally with a master and a worker.

curl -s http://127.0.0.1:8080/a.html http://127.0.0.1:8080/b.html

[debug] 11#11: posix_memalign: 0000559582B79BC0:512 @16
[debug] 11#11: *2 accept: 127.0.0.1:41964 fd:3
[debug] 11#11: *2 malloc: 0000559582B7C410:1024
[debug] 11#11: *2 posix_memalign: 0000559582B76700:4096 @16
[debug] 11#11: *2 http request line: "GET /a.html HTTP/1.1"
[debug] 11#11: *2 posix_memalign: 0000559582BA3320:4096 @16
[debug] 11#11: *2 add cleanup: 0000559582B77450
[debug] 11#11: *2 set http keepalive handler
[debug] 11#11: *2 http close request
[debug] 11#11: *2 run cleanup: 0000559582B77450
[debug] 11#11: *2 file cleanup: fd:10
[debug] 11#11: *2 free: 0000559582B76700, unused: 16
[debug] 11#11: *2 free: 0000559582BA3320, unused: 2608
[debug] 11#11: *2 free: 0000559582B7C410
[debug] 11#11: *2 hc free: 0000000000000000
[debug] 11#11: *2 hc busy: 0000000000000000 0
[debug] 11#11: *2 http keepalive handler
[debug] 11#11: *2 malloc: 0000559582B7C410:1024
[debug] 11#11: *2 posix_memalign: 0000559582B76700:4096 @16
[debug] 11#11: *2 http request line: "GET /b.html HTTP/1.1"
...
[debug] 11#11: *2 http keepalive handler
[debug] 11#11: *2 malloc: 0000559582B7C410:1024
[debug] 11#11: *2 close http connection: 3
[debug] 11#11: *2 free: 0000559582B7C410
[debug] 11#11: *2 free: 0000559582B79BC0, unused: 128

(The date and time at the start of each line were removed. The pid 11, connection number *2, addresses and fd differ on every run. The lines for the b.html request were identical to a.html, addresses and all, so they are shortened to ....)

Matching them to the source from the top:

Three things are confirmed.

• A request pool is created and destroyed for every request. On the same connection (*2), a.html and b.html each start with posix_memalign …4096 and end with free: …, unused:. The connection pool (…79BC0) disappears only once, when the connection closes.
• The second request got the same addresses as the first (…76700, …BA3320, …C410). The pool hands free()d memory back to the C library's malloc, and malloc gives the spot it just got back to the next request of the same size. Section 1's "reuse" is visible in real addresses.
• Cleanups run before blocks are given back. The run cleanup/file cleanup lines come before the free: lines. The cleanup function needs to read data inside the pool (clnf->fd), so it has to be this order.

The malloc: …:1024 right before *2 close http connection is the trace of the keep-alive handler taking a buffer again to read from as curl closed the connection, reading 0 bytes, and closing it. When the worker exits (-s quit), lines follow showing the cycle pool being torn down the same way. The cycle pool had five 16KB blocks and seven large allocations (output not shown).

Easy to get wrong

• ngx_reset_pool() does not run cleanups. There is a separate function that empties a pool. It free()s the large allocations and rewinds every block's last to just past the header (ngx_palloc.c lines 99–119), but it leaves the cleanup list alone. In this commit it is called only inside the HTTP and stream geo modules, so you will hardly meet it in the request-handling flow.
• When rewinding, it uses sizeof(ngx_pool_t) even from the second block on. ngx_reset_pool() sets every block's last back to p + sizeof(ngx_pool_t), so blocks that only used a 32-byte ngx_pool_data_t header end up 48 bytes shorter. That's simplicity rather than a bug.
• r->cleanup is a different list from the pool's cleanups. Before destroying the pool, ngx_http_free_request() first runs the request's own cleanup list r->cleanup (lines 3999–4008). Upstream uses it to tidy up the backend connection (Part 11).

5. ngx_str_t — strings that don't end in NUL

typedef struct {
    size_t      len;
    u_char     *data;
} ngx_str_t;

src/core/ngx_string.h lines 16–19

A length and a start address — that's all. A C string marks its length with a '\0' at the end, but ngx_str_t carries its length separately, so it doesn't need a '\0' at the end. Why that matters becomes clear when you think about the request a client sends. Having received the request line GET /index.html HTTP/1.1, nginx doesn't copy the URI into new memory; it records just a data pointing at the / inside the header buffer, and a length of 11. No copy, no allocation. With C strings it would have had to write a '\0' at the end of the URI (where the space is) or make a copy.

The rest of Example 1: r->uri seen in gdb

At the same breakpoint in Example 1 we also printed the ngx_str_ts of a request that carried a Host: example.test header.

=== 1. ngx_str_t is not NUL-terminated ===
$1 = {len = 11, data = 0x555555783414 "/index.html HTTP/1.1\r\nHost"}
$2 = 11
$3 = {len = 12, data = 0x555555783430 "example.test"}

gdb treats a u_char * as a C string and prints until it finds a '\0'. So even though r->uri has length 11, the HTTP/1.1\r\nHost after it shows up too. That's proof that the URI points straight into the header buffer. But why did it stop at Host? There should be a : there.

            h = ngx_list_push(&r->headers_in.headers);
            if (h == NULL) {
                ngx_http_close_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR);
                break;
            }

            h->hash = r->header_hash;

            h->key.len = r->header_name_end - r->header_name_start;
            h->key.data = r->header_name_start;
            h->key.data[h->key.len] = '\0';

            h->value.len = r->header_end - r->header_start;
            h->value.data = r->header_start;
            h->value.data[h->value.len] = '\0';

src/http/ngx_http_request.c lines 1512–1526

ngx_http_process_request_headers(), which reads the headers, writes a '\0' right after the header name (where the : was) and right after the value (where the \r was). By overwriting bytes in the buffer, it makes header names and values usable as C strings too. That's why $3 (the Host value) stopped exactly at example.test. The URI in the request line gets no such treatment.

Three ngx_str_ts all point into one buffer. The two red cells are bytes nginx overwrote.

Tools for working with strings

nginx's logging functions are all in the ngx_sprintf family, so all over the source you'll see ngx_str_ts printed with %V, as in "invalid server name or wildcard \"%V\" on %V". ngx_http.c line 1646, which builds the list of server_name names, is one example.

Easy to get wrong

• Don't pass an ngx_str_t's data to strcmp(), printf("%s") or strlen(). As in the gdb output above, they read on past the end of the string until they hit a '\0'. Compare with functions that take a length, like ngx_strncmp(a.data, b.data, len). Header names and values happen to end in '\0' (really, thanks to the code above), but that is a favour from the header-reading code, not a promise of ngx_str_t.
• Passing a pointer variable to ngx_string() is wrong. It measures the length with sizeof, so passing a char *p gives a length of the pointer size (8) minus 1, i.e. 7. Use it only with string literals "...".
• An ngx_str_t that points into a buffer can't outlive that buffer. URIs and header values point into the header buffer. If a value must survive after the request ends, ngx_pstrdup() it into a longer-lived pool. This ties back to the keep-alive ngx_pfree() in section 4 — that buffer really is given back when the request ends.

6. Arrays, lists, queues, hashes and red-black trees

nginx builds and uses its own data structures that the C standard library doesn't provide. They all have one thing in common — they take their memory from a pool — so there are hardly any functions to "destroy" them.

The grey box is the old chunk: nobody points to it, but it occupies memory until the pool is destroyed.

ngx_array_t — a growable array

typedef struct {
    void        *elts;
    ngx_uint_t   nelts;
    size_t       size;
    ngx_uint_t   nalloc;
    ngx_pool_t  *pool;
} ngx_array_t;

src/core/ngx_array.h lines 16–22

elts is the memory where the elements sit side by side, nelts is the current number of elements, size is the size of one element, and nalloc is the number of slots. Since it doesn't know the element type, it works with void * and size, and the caller casts elts to a pointer of its own type. Here is the "array is full" part of the function that adds one element:

    if (a->nelts == a->nalloc) {

        /* the array is full */

        size = a->size * a->nalloc;

        p = a->pool;

        if ((u_char *) a->elts + size == p->d.last
            && p->d.last + a->size <= p->d.end)
        {
            /*
             * the array allocation is the last in the pool
             * and there is space for new allocation
             */

            p->d.last += a->size;
            a->nalloc++;

        } else {
            /* allocate a new array */

            new = ngx_palloc(p, 2 * size);
            if (new == NULL) {
                return NULL;
            }

            ngx_memcpy(new, a->elts, size);
            a->elts = new;
            a->nalloc *= 2;
        }
    }

src/core/ngx_array.c lines 54–85

Only someone who knows the pool's pointer bump could write this. If the array is the last thing allocated from the pool's first block and there is room after it, it bumps d.last by one more element and grows in place. Otherwise it takes a new chunk twice the size and copies. The old chunk can't be given back, so it stays in the pool (the grey box in the figure above).

Connecting it to the config: the phase handler lists of Parts 2 and 9 are such arrays. After the config is read, the static module inserts its handler with h = ngx_array_push(&cmcf->phases[NGX_HTTP_CONTENT_PHASE].handlers); (ngx_http_static_module.c line 289). The names of a server_name and the listen addresses are arrays too.

ngx_list_t — a list whose elements never move

struct ngx_list_part_s {
    void             *elts;
    ngx_uint_t        nelts;
    ngx_list_part_t  *next;
};


typedef struct {
    ngx_list_part_t  *last;
    ngx_list_part_t   part;
    size_t            size;
    ngx_uint_t        nalloc;
    ngx_pool_t       *pool;
} ngx_list_t;

src/core/ngx_list.h lines 18–31

It is a chain of array chunks (parts). When the last part fills up, ngx_list_push() takes one more part of the same size and appends it (ngx_list.c lines 30–63). Since nothing is copied, an element's address never changes once it is in.

That is why request headers are a list rather than an array. In section 5's code, a header line went in via ngx_list_push(&r->headers_in.headers) (headers is an ngx_list_t, ngx_http_request.h line 187), and fields such as r->headers_in.host point by pointer to an element inside that list (line 190). With an array, a copy triggered by more headers would leave those pointers aimed at the old chunk. The response headers headers_out.headers are a list for the same reason, created with 20 slots up front in section 1's ngx_http_alloc_request() (lines 609–615).

How to walk a list is written as a comment in the header file (when i inside a part reaches nelts, move to the next part, ngx_list.h lines 55–77). You'll see exactly that shape in module code that walks headers.

ngx_queue_t — a doubly linked ring embedded in your struct

struct ngx_queue_s {
    ngx_queue_t  *prev;
    ngx_queue_t  *next;
};

src/core/ngx_queue.h lines 18–21

There is no data field. Instead, you put an ngx_queue_t field inside the struct you want to put on the ring (often called an "intrusive list"). For example, ngx_connection_t has an ngx_queue_t queue; field (ngx_connection.h line 168). This macro recovers the outer struct from an ngx_queue_t * taken off the ring:

#define ngx_queue_data(q, type, link)                                         \
    (type *) ((u_char *) q - offsetof(type, link))

src/core/ngx_queue.h lines 106–107

offsetof(type, link) is the number of bytes from the start of the struct to that field. Subtract it from the field's address and you get the struct's start address. ngx_drain_connections(), which picks idle keep-alive connections to close when connections run short, uses it like this:

        q = ngx_queue_last(&cycle->reusable_connections_queue);
        c = ngx_queue_data(q, ngx_connection_t, queue);

src/core/ngx_connection.c lines 1434–1435

Inserting and removing (ngx_queue_insert_head, ngx_queue_remove …) are all macros that change a few pointers, with no allocation at all. Part 12's upstream keepalive cache also puts connections on and takes them off such a queue.

ngx_hash_t — a hash built once at config time and only read after

Hash tables are used to find a virtual server by server_name, to look up variable names, and for map. nginx's hash differs in character from a typical hash table: it collects all the keys at config-read time, builds the table in one go, and never inserts or removes after that. Here is the lookup function:

void *
ngx_hash_find(ngx_hash_t *hash, ngx_uint_t key, u_char *name, size_t len)
{
    ngx_uint_t       i;
    ngx_hash_elt_t  *elt;

    /* ... */

    elt = hash->buckets[key % hash->size];

    if (elt == NULL) {
        return NULL;
    }

    while (elt->value) {
        if (len != (size_t) elt->len) {
            goto next;
        }

        for (i = 0; i < len; i++) {
            if (name[i] != elt->name[i]) {
                goto next;
            }
        }

        return elt->value;

    next:

        elt = (ngx_hash_elt_t *) ngx_align_ptr(&elt->name[0] + elt->len,
                                               sizeof(void *));
        continue;
    }

    return NULL;
}

src/core/ngx_hash.c lines 12–49

The elements in one bucket (the slot where entries with the same hash value gather) are not a linked list but laid out back to back in memory. One element is a value pointer, a len and the name's characters (ngx_hash.h lines 16–20), and the next element starts where the end of the name is rounded up to 8 bytes. A slot with value == NULL marks the end of the bucket. No jumping around memory following pointers, so it plays well with the CPU cache.

The price is an upper limit on "bytes per bucket". This is the error everyone who runs nginx meets at some point. The bucket size is server_names_hash_bucket_size, whose default is the CPU cache line size (ngx_cacheline_size) (ngx_http_core_module.c lines 3549–3554). ngx_hash_init(), which builds the hash, first checks that each name fits in one bucket:

#define NGX_HASH_ELT_SIZE(name)                                               \
    (sizeof(void *) + ngx_align((name)->key.len + 2, sizeof(void *)))

src/core/ngx_hash.c lines 248–249

        if (hinit->bucket_size < NGX_HASH_ELT_SIZE(&names[n]) + sizeof(void *))
        {
            ngx_log_error(NGX_LOG_EMERG, hinit->pool->log, 0,
                          "could not build %s, you should "
                          "increase %s_bucket_size: %i",
                          hinit->name, hinit->name, hinit->bucket_size);
            return NGX_ERROR;
        }

src/core/ngx_hash.c lines 281–288

An element is the 8-byte value pointer plus (the name length + the 2-byte len, rounded up to a multiple of 8), and the bucket also needs the 8-byte NULL slot at its end. With a 64-byte bucket that's 8 + align(len + 2, 8) + 8 ≤ 64, i.e. names up to 46 characters. We ran it to check the arithmetic.

Counting in bytes whether one name fits in a bucket. The 64 is the bucket size from Example 3's output below.

Example 3: a 46-character and a 47-character server_name

A script that runs nginx -t while changing only the length of a server_name (ex/u03-hash-bucket/run.sh). It tries the second server {} with a name of 33 as + .example.test (46 characters) and of 34 as + .example.test (47 characters), and finally adds server_names_hash_bucket_size 128; with the 47-character name.

=== server_name of 46 chars ===
nginx: the configuration file /tmp/u03h/nginx.conf syntax is ok
nginx: configuration file /tmp/u03h/nginx.conf test is successful
=== server_name of 47 chars ===
nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
nginx: configuration file /tmp/u03h/nginx.conf test failed
=== same 47-char name with server_names_hash_bucket_size 128 ===
nginx: the configuration file /tmp/u03h/nginx.conf syntax is ok
nginx: configuration file /tmp/u03h/nginx.conf test is successful

Just as computed, 46 characters pass and 47 produce exactly the error sentence from the source above. The 64 at the end of the error is this container's cache line size. Why the common advice "increase server_names_hash_bucket_size" works is all in that one line. The "could not build optimal …" warning you get when the number of buckets (server_names_hash_max_size, default 512) runs out lives in the same function (ngx_hash.c lines 337–344).

Wildcard names like *.example.com go into separate wc_head·wc_tail hashes, and at lookup time ngx_hash_find_combined() tries the exact name → leading wildcard → trailing wildcard, in that order (ngx_hash.c lines 211–245). A wildcard hash is built in layers, splitting the name at dots (.), and hides "is this value data or the next layer's hash?" in the low two bits of the value pointer (the comment at lines 90–99). Finding the virtual server from a request's Host is covered in Part 8.

The building side also shows a pool idiom. ngx_http_server_names(), which builds the server_name hash, creates a temporary pool ha.temp_pool used only while building (ngx_http.c line 1611), leaves the finished hash in cf->pool (the cycle pool), and destroys only the temporary pool (line 1710). "Results go in the long-lived pool, scraps from the work in the short-lived one" is the basic pattern of pool use. The conf.temp_pool that ngx_init_cycle() uses while reading the config follows the same idea (ngx_cycle.c line 262, line 781).

ngx_rbtree_t — the red-black tree used for timers

A red-black tree is a self-balancing binary search tree in which inserting, removing and finding the smallest value all take time proportional to the logarithm of the number of elements. nginx's is intrusive, like the queue. A node, ngx_rbtree_node_t, is a small struct with key, left, right, parent, color and data (lines 22–29), and the tree looks like this:

typedef struct ngx_rbtree_s  ngx_rbtree_t;

typedef void (*ngx_rbtree_insert_pt) (ngx_rbtree_node_t *root,
    ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel);

struct ngx_rbtree_s {
    ngx_rbtree_node_t     *root;
    ngx_rbtree_node_t     *sentinel;
    ngx_rbtree_insert_pt   insert;
};

src/core/ngx_rbtree.h lines 32–41

Its distinctive feature is that each tree takes the function that picks the insertion spot, insert, as a function pointer. Balancing is shared; only the comparison, such as "how to treat equal keys", is decided by the user. The biggest user is the event timers. ngx_event_t embeds an ngx_rbtree_node_t timer; (ngx_event.h line 97), and when handling expired timers the outer event is recovered with ev = ngx_rbtree_data(node, ngx_event_t, timer); (ngx_event_timer.c line 76). The very same offsetof trick as ngx_queue_data. How timers are used in the event loop is in Part 7.

A taste: memory shared between workers — slab

All the pools and data structures so far live inside one worker process. When workers need to share state — limit_req_zone, proxy_cache_path … keys_zone=, upstream { zone … } — they use shared memory, and the thing that carves it up is ngx_slab_pool_t (src/core/ngx_slab.c). Unlike a pool, it has ngx_slab_free() to give back a single piece, and since several processes use it at once, it takes a lock on every allocation — ngx_slab_alloc() does nothing but call ngx_slab_alloc_locked() between ngx_shmtx_lock() and ngx_shmtx_unlock() (ngx_slab.c lines 169–180).

The pool with "no locks, no per-piece freeing" and the slab with "locks, per-piece freeing" — different jobs, opposite designs. This post doesn't go into how the slab manages pages and size classes.

7. ngx_buf_t and ngx_chain_t — the containers that carry a response

Finally, the structs that response data travels out on.

struct ngx_buf_s {
    u_char          *pos;
    u_char          *last;
    off_t            file_pos;
    off_t            file_last;

    u_char          *start;         /* start of buffer */
    u_char          *end;           /* end of buffer */
    ngx_buf_tag_t    tag;
    ngx_file_t      *file;
    ngx_buf_t       *shadow;


    /* the buf's content could be changed */
    unsigned         temporary:1;

    /*
     * the buf's content is in a memory cache or in a read only memory
     * and must not be changed
     */
    unsigned         memory:1;

    /* the buf's content is mmap()ed and must not be changed */
    unsigned         mmap:1;

    unsigned         recycled:1;
    unsigned         in_file:1;
    unsigned         flush:1;
    unsigned         sync:1;
    unsigned         last_buf:1;
    unsigned         last_in_chain:1;

    unsigned         last_shadow:1;
    unsigned         temp_file:1;

    /* STUB */ int   num;
};


struct ngx_chain_s {
    ngx_buf_t    *buf;
    ngx_chain_t  *next;
};

src/core/ngx_buf.h lines 20–62

The key point is that one buffer can be memory or a file.

• Memory buffer — start·end span the whole piece of memory, and pos·last the part not yet sent (or not yet read). If the contents may be changed, the temporary bit is set; if it's read-only (a config string, say) and must not change, memory is set.
• File buffer — in_file is set, and file (fd and name) and file_pos·file_last (a position inside the file) are recorded. It carries only "this file, from byte X to byte Y" without reading the data into memory. Whoever receives it at the end can have the kernel send it directly with sendfile().
• Marker bits — last_buf means "the last buffer of this response", flush "don't hold on to it, send now", and last_in_chain "the last one in this chain".

: 1 is a C bit field: the field takes up only one bit. Eleven flags fit into a single unsigned. There are macros that read the bits together, too — for example ngx_buf_size(b) is last - pos for a memory buffer and file_last - file_pos otherwise (ngx_buf.h lines 125–138).

ngx_chain_t is a singly linked list with just two fields: one buffer and the next link. The end of the static-file handler, where it builds the response body, shows at a glance how the two are used.

    b->file_pos = 0;
    b->file_last = of.size;

    b->in_file = b->file_last ? 1 : 0;
    b->last_buf = (r == r->main) ? 1 : 0;
    b->last_in_chain = 1;
    b->sync = (b->last_buf || b->in_file) ? 0 : 1;

    b->file->fd = of.fd;
    b->file->name = path;
    b->file->log = log;
    b->file->directio = of.is_directio;

    out.buf = b;
    out.next = NULL;

    return ngx_http_output_filter(r, &out);

src/http/modules/ngx_http_static_module.c lines 261–277

Not a single byte of the file has been read. The buffer came from the request pool via ngx_calloc_buf(r->pool) (line 245), and one file buffer pointing at the whole file (from 0 to of.size) is put on one link (out) and handed to the filter chain. The link out is not even from a pool — it is a local variable of the function (on the stack). A filter that needs to hold on to it copies it into a link of its own. The file fd opened here is closed along with the request pool by the cleanup we saw in section 4.

Response headers (memory) and the file body (a file) can ride on the same chain.

Used links go back to the pool's chain

The chain field of ngx_pool_t that we skipped in section 2 is used here. Filters create and discard countless links while sending one request; taking each 16-byte link fresh from the pool would fill blocks quickly. So used links are hung on the pool's chain list and taken back out next time. ngx_alloc_chain_link(), which hands out links, takes the first one off pool->chain if there is one and calls ngx_palloc() only when there isn't (src/core/ngx_buf.c lines 47–65)

The returning side, ngx_free_chain(pool, cl), is a two-line macro that slips the link onto the front of pool->chain (ngx_buf.h lines 148–150). The keep-alive code in section 4 used it too, when tidying up header-buffer links (ngx_free_chain(c->pool, ln), ngx_http_request.c line 3494). In the same spirit, filters recycle used buffers through free·busy lists (ngx_chain_update_chains()).

How the filter chain takes this chain, adds headers, turns it into gzip, and finally sends it with writev·sendfile is in Part 10.

Easy to get wrong

• pos/last and start/end are different things. start/end are the boundaries of the memory piece and don't change; pos/last move as data is sent or read. The amount to send is last - pos, not end - start.
• The memory bits and in_file are not mutually exclusive. The macros are written on the assumption that a buffer can have both set (the same contents in memory and in a file). The trace of that is ngx_buf_in_memory_only(), which asks "is it only in memory?" by checking !(b)->in_file separately from ngx_buf_in_memory() (ngx_buf.h line 126).

8. Compared with httpd's APR pools

Put side by side with the APR pool from httpd Part 4, you can see how differently the same idea was implemented. The APR column is what httpd Part 4 confirmed against its source.

nginx's side is much simpler. ngx_palloc.c is 430 lines, with node reuse, the tree and thread provisions all left out. It seems to be because each worker runs an event loop on a single thread (Part 2), so it doesn't have to cope with many threads and many MPMs the way httpd does (the design intent isn't written in the source, so this is a guess). Without a reuse list, it gives blocks back to the C library's malloc and relies on malloc handing the same spot out again, as we saw in Example 2.

The table's "Relationship between pools" and "Where used memory goes" rows as a picture. The APR side is what httpd Part 4 confirmed; the address is from Example 2's output.

What this post leaves out

To keep to the main thread, these get only a name and a location.

• ngx_pmemalign() — allocation with an alignment of your choice; always goes to large. Used, for example, in ngx_output_chain.c to get Direct I/O buffers (ngx_palloc.c lines 252–274).
• ngx_pool_run_cleanup_file()·ngx_pool_delete_file() — running only one fd's cleanup before the pool is destroyed, and closing a file while deleting it (ngx_palloc.c lines 342–401).
• ngx_create_chain_of_bufs()·ngx_chain_update_chains()·ngx_chain_get_free_buf() — creating bundles of buffers and managing the free/busy lists (src/core/ngx_buf.c). Used by the filters and the event pipe in Parts 10 and 11.
• ngx_radix_tree.c — a radix tree for looking up IP address ranges, used by the geo module.
• Building hashes with ngx_hash_add_key()·ngx_hash_wildcard_init() — collecting keys and building wildcard layers (from ngx_hash.c line 490).
• Balancing the red-black tree — the rotations in ngx_rbtree_insert()·ngx_rbtree_delete() (src/core/ngx_rbtree.c).
• Slab internals — page-level management and size classes (src/core/ngx_slab.c), and ngx_init_zone_pool(), which sets up a shared memory zone (from ngx_cycle.c line 966).
• The old cycle's pool on reload — the path where, outside the master, the old cycle can't be thrown away immediately and is collected in ngx_old_cycles and cleared by a 30-second timer (ngx_cycle.c lines 792–830). Continued in Part 4.

The pool and data-structure files in this post (ngx_palloc.[ch], ngx_array.c, ngx_list.c, ngx_hash.c, ngx_buf.h, ngx_string.h, ngx_queue.h) have no changed lines between the stable release 1.30.6 and this commit. If you are on 1.30.x, everything in this post still holds.

Summary

What's next

Next we go to where the thing this post only called "the cycle pool" is created. How main() reads the command line, how ngx_init_cycle() creates a fresh pool and loads the config file onto it, and how the same function is called again on reload to stand a new cycle next to the old one continues in Part 4: Startup and Reading the Config — From main() to ngx_init_cycle. We'll also see ngx_conf_read_token(), which cuts nginx.conf into tokens.

• Part 1: How I Read It with AI, and the Map
• Part 2: The Big Picture — Master and Workers, Modules, the Cycle and Config Contexts
• Part 7: The Event Loop — epoll, Timers, Posted Events and accept — where this post's red-black tree is used for timers
• Part 10: Filters and Static Files — How a Response Flows Out — the path ngx_chain_t takes through the filters
• Posts in the httpd series on the same problems: httpd Part 4: APR Memory Pools — Managing Memory Without free(), and a different answer to the response container, httpd Part 11: Filters and Bucket Brigades