Digging into Apache httpd's Source 4: APR Memory Pools — Managing Memory Without free()

What this post covers

This is the fourth post in the "Digging into Apache httpd's Source" series. Part 3 showed that httpd is built in layers — OS ↔ APR ↔ core ↔ MPM ↔ modules — and that the three big structs, server_rec, conn_rec and request_rec, each have a field called pool. This time we open up that pool: the APR memory pool.

The first time you open httpd's source, something looks odd. It is a C program, yet malloc() and free() are almost nowhere to be seen. Instead there are calls like apr_palloc(r->pool, …) everywhere, and no code ever gives that memory back. This post follows the reason why.

There are four questions to answer.

• If memory is only ever taken and never handed back, why doesn't it leak?
• Why is apr_palloc() faster than malloc() — what does it actually do inside?
• Things that aren't memory, like files and sockets — when are they closed, and in what order?
• Which pools does httpd create, when are they thrown away, and what should you watch out for when writing a module?

Every line number in this post refers to those two commits. Part 1 explains how to get the httpd source at the same commit and the Docker setup used to run the examples.

With all its debugging branches, the pool code is close to 3,000 lines, but this post keeps only the main thread that someone opening the source for the first time can follow. What was left out is named at the end, under "What this post leaves out".

Words to know first

malloc() and free() are the memory functions of the C standard library. malloc(100) lends you 100 bytes and returns their address. When you're done, you must call free() with that address to give it back. If you don't, that memory can't be used again until the program exits. That is a memory leak. Going the other way, if you keep using an address after giving it back, you're touching memory that may already be in use for something else (use-after-free).

A pool is a "bucket" that lends memory. Memory taken from a pool is never handed back piece by piece; it all goes back in one go when the pool is cleared (apr_pool_clear()) or destroyed (apr_pool_destroy()). APR's type name for it is apr_pool_t.

Lifetime is how long a piece of data is in use. Request headers are useless once the request is over; configuration is needed until the server restarts. Think of a pool as a way to put data with the same lifetime into one bucket.

A node is a big chunk of memory the pool actually stores things in. The pool fetches chunks in multiples of 4KB and hands out small slices of them. In the source it is apr_memnode_t.

The allocator is the manager that makes nodes and takes them back. It doesn't give finished nodes straight back to the operating system; it keeps them in lists sorted by size and hands them out again later. In the source it is apr_allocator_t.

A cleanup is a function you leave with a pool, asking it to be called when the pool is cleared or destroyed. Tidying up that isn't "returning memory" — closing a file, closing a socket — goes here. It uses a function pointer (passing a function around like a value) and void * (a pointer that can point to any type). Both are explained in Part 2.

1. Why there's no free — a server's memory dies in batches

Picture the data a web server creates while handling one request: strings for the request line and headers, the URL turned into a file path, the table of response headers, per-request config fragments for each module… Dozens to hundreds of small pieces of memory. And nearly all of them stop being needed at the same moment the request ends.

Managing this with malloc()/free() means deciding, for every single piece, "who calls free(), and when". In httpd, where dozens of modules plug in, that is hard to get right every time. Miss one spot and memory leaks on every request — and since a server runs for weeks, it eventually eats all the memory.

APR pools change the question. Instead of "when do I throw this piece away", you only decide "what does this piece die with".

• Dies with the request → take it from the request pool, r->pool
• Dies with the connection → take it from the connection pool, c->pool
• Dies with the configuration → take it from the config pool, pconf

Then, when a request ends, httpd's core destroys r->pool once, and all the memory taken during that request goes back. Modules never need to call free() — and they have no way to. APR has no function that returns one piece taken from a pool. The allocation functions in apr_pools.h are just apr_palloc() and apr_pcalloc(), and the only functions that give memory back, apr_pool_clear() and apr_pool_destroy(), work on the whole pool (include/apr_pools.h lines 357–483).

The question changes from "when do I throw this away" to "what does this die with". The red box is the leak malloc/free makes easy.

You can see a trace of this design in the config file too. The MaxMemFree directive sets "how much memory the pools may keep hold of instead of handing it back to the operating system". The allocator in section 2 is exactly where that memory is kept.

2. What a pool looks like — nodes and the allocator

The pool struct

Here is the start of apr_pool_t. (/* ... */ marks what this post cut; everything else is the source as-is.)

struct apr_pool_t {
    apr_pool_t           *parent;
    apr_pool_t           *child;
    apr_pool_t           *sibling;
    apr_pool_t          **ref;
    cleanup_t            *cleanups;
    cleanup_t            *free_cleanups;
    apr_allocator_t      *allocator;
    struct process_chain *subprocesses;
    apr_abortfunc_t       abort_fn;
    apr_hash_t           *user_data;
    const char           *tag;

#if !APR_POOL_DEBUG
    apr_memnode_t        *active;
    apr_memnode_t        *self; /* The node containing the pool itself */
    char                 *self_first_avail;
    /* ... */

memory/unix/apr_pools.c lines 562–578

Read it as three groups.

• parent, child, sibling — pools form a tree. When you create a pool you name its parent (apr_pool_create(&newpool, parent)), and the new pool is attached to the front of the parent's child list (lines 1106–1116). When the parent is cleared or destroyed, its children go first. That is the heart of section 4.
• cleanups — the list of tidy-up functions from section 1. Section 4 covers it.
• active, self — the nodes that actually hold the memory. self is the node that holds the pool struct itself; active is the node currently handing out memory.

#if !APR_POOL_DEBUG is a preprocessor instruction meaning "include these fields only when APR is not built in pool-debugging mode". In debugging mode the pool works in a completely different way; that story is at the end of section 3.

As the comment on self says, the pool struct lives inside its own first node. Creating a pool first fetches a node, places the apr_pool_t at the very front of it, and uses the rest as memory to hand out (lines 1086–1092). So even creating a pool costs one node. The minimum node size, MIN_ALLOC, is 2 * BOUNDARY_SIZE, and in a default build BOUNDARY_SIZE is 1 << 12 = 4096, so that's 8KB (line 85, lines 94–95). If APR is built to get its nodes with mmap, the value follows the page size instead (lines 88–92); on ordinary x86-64 Linux the page size is 4KB, so the result is the same.

Nodes

The struct at the head of each node has just six fields.

struct apr_memnode_t {
    apr_memnode_t *next;            /**< next memnode */
    apr_memnode_t **ref;            /**< reference to self */
    apr_uint32_t   index;           /**< size */
    apr_uint32_t   free_index;      /**< how much free */
    char          *first_avail;     /**< pointer to first free memory */
    char          *endp;            /**< pointer to end of free memory */
};

include/apr_allocator.h lines 54–61

The two lines that matter are the last two. first_avail is the first byte not yet given to anyone, and endp is the end of the node. The space between them is what's left. Nowhere does the node record where each slice already handed out starts and ends. It doesn't need to — slices are never handed back one at a time.

Yellow is memory; green is the free space the next apr_palloc() will slice from.

A pool's nodes form a ring (a circular list) linked by next. Every time apr_palloc() slots in a new node, it keeps the ring ordered by how much free space each node has (lines 861–875).

The allocator — where finished nodes are kept

When a pool is cleared, its nodes are not free()d; they go back to the allocator. The allocator keeps nodes in lists by size, free[].

struct apr_allocator_t {
    /* ... */
    /**
     * Lists of free nodes. Slot 0 is used for oversized nodes,
     * and the slots 1..MAX_INDEX-1 contain nodes of sizes
     * (i+1) * BOUNDARY_SIZE. Example for BOUNDARY_INDEX == 12:
     * slot  0: nodes larger than 81920
     * slot  1: size  8192
     * slot  2: size 12288
     * ...
     * slot 19: size 81920
     */
    apr_memnode_t      *free[MAX_INDEX];
};

memory/unix/apr_pools.c lines 125–154

The next time a pool asks for a node, the allocator first looks for the right size in these lists, and calls malloc() only when there isn't one (allocator_alloc() lines 289–424; the malloc is on line 402). In a server that needs nodes of the same sizes for every request, most of the time it ends with taking one off a list.

But if it only ever kept nodes, then after one huge request it would hold on to that memory forever. So the allocator has a "maximum it may keep", max_free_index, and any node beyond that is free()d the moment it comes back (lines 448–452; the free() is at lines 485–494). The default is APR_ALLOCATOR_MAX_FREE_UNLIMITED (0) — "keep everything" (apr_allocator_create() line 191).

httpd sets this value with the MaxMemFree directive. When the directive is read, ap_max_mem_free gets KB × 1024 (server/mpm_common.c line 471), and if you don't set it the default is 2048*1024, i.e. 2MB (lines 186–187). The MPM puts this value in each allocator it creates, per connection or per child process (section 5).

Easy to get confused

• Clearing a pool may not lower the process's memory use. Nodes only go back to the allocator, and as long as it's under max_free_index they aren't free()d. When top shows memory staying put after requests finish, that's usually this, not a leak. The next request reuses those nodes, though, so it doesn't keep growing.
• Even free() doesn't necessarily return memory to the operating system right away. That's up to the C library's malloc implementation and outside APR's source.

3. apr_palloc — allocating is pushing a pointer

Now the function that takes memory from a pool. Here is the start of apr_palloc(), with the Valgrind (memory-checking tool) branch and the size-overflow error handling cut.

APR_DECLARE(void *) apr_palloc(apr_pool_t *pool, apr_size_t in_size)
{
    apr_memnode_t *active, *node;
    void *mem;
    apr_size_t size, free_index;

    pool_concurrency_set_used(pool);
    size = APR_ALIGN_DEFAULT(in_size);
    /* ... */
    active = pool->active;

    /* If the active node has enough bytes left, use it. */
    if (size <= node_free_space(active)) {
        mem = active->first_avail;
        active->first_avail += size;
        goto have_mem;
    }
    /* ... */

memory/unix/apr_pools.c lines 810–836

1. Round the size up to a multiple of 8. APR_ALIGN_DEFAULT(size) is APR_ALIGN(size, 8), and APR_ALIGN is a macro that uses bit operations to get the next multiple of 8 (include/apr_general.h lines 136–140). Ask for 13 bytes and 16 are used. The next slice's address is then also a multiple of 8, so whatever type you store there sits at a position the CPU likes to read from (alignment).
2. If the current node has enough space left — node_free_space() is a macro that computes endp - first_avail (line 742).
3. Return first_avail, then push first_avail forward by the size. That's it.

The APR_DECLARE(void *) in front of the function name is a macro meaning "export this function from the library; it returns void *", and goto have_mem jumps straight to the have_mem: label at the end of the function.

One addition and one comparison and the allocation is done. malloc() has to find free space, record each slice's size for a later free(), and cope with several threads calling it at once. The pool does none of that. Since slices are never handed back one at a time, it needn't record sizes, nor look for returned gaps to reuse. This style is commonly called bump pointer allocation (the term itself isn't in the source).

Only when the current node is full does a bit more work happen.

The green path is the one taken almost every time. malloc() can only be called in the yellow box.

It checks the very next node in the ring (the one with the most free space), and if that's not enough either, it gets a new node from the allocator (lines 838–850). Either way, that node becomes active and the pointer is pushed in the same way (lines 852–859). Nodes come in 4KB multiples (allocator_align() lines 263–279), so asking for 100KB at once just creates one node that big; the method is the same.

Example 1: is it really just a pointer push?

A small C program that uses a pool directly. It doesn't use httpd at all; it only links the APR library (libapr-1). Every example in this post was run in the same container (Ubuntu 24.04, packaged APR 1.7.2) as httpd 2.5.1-dev built unmodified from the reference commit (e4af30fe6d). The whole program is pooldemo.c in the example folder; only part 1 is shown here.

/* An example written for this post (not httpd source) */
    apr_initialize();
    apr_pool_create(&pool, NULL);

    /* 1. Allocating is pushing a pointer forward */
    a = apr_palloc(pool, 13);
    b = apr_palloc(pool, 13);
    c = apr_palloc(pool, 100);
    printf("[1] palloc(13), palloc(13), palloc(100)\n");
    printf("  b - a = %ld bytes, c - b = %ld bytes\n", (long)(b - a), (long)(c - b));

Compiled and run inside the container like this:

gcc $(apr-1-config --cflags --cppflags --includes) -o /tmp/pooldemo pooldemo.c $(apr-1-config --link-ld)
/tmp/pooldemo

[1] palloc(13), palloc(13), palloc(100)
  b - a = 16 bytes, c - b = 16 bytes

The two 13-byte slices came out right next to each other, 16 bytes apart. The pointer moved by exactly the size rounded up to a multiple of 8. With malloc(), each slice usually carries a header such as its size, so the gap would normally be bigger.

Easy to get confused

• Memory from apr_palloc() is not zeroed. Nodes are reused, so whatever an earlier request left there may still be there. If you need zeroed memory, use apr_pcalloc(). In a build without pool debugging, apr_pcalloc is a macro that becomes memset(apr_palloc(p, size), 0, size) (include/apr_pools.h lines 464–465). That's also why ap_create_request() gets its request_rec with apr_pcalloc() (section 5).
• apr_palloc() takes no lock. There's no mutex on the fast path above; a lock is only taken when a node is pulled from the allocator (allocator_alloc() line 316). The pool_concurrency_set_used() on the first line only does something when APR is built with APR_POOL_CONCURRENCY_CHECK — it aborts if another thread is using the same pool at the same time — and is otherwise an empty function (lines 744–804). So the rule is one pool, one thread. The header does say that creating child pools under one parent from several threads at once is safe (include/apr_pools.h lines 191–194).
• None of this holds in pool-debugging mode. In APR built with APR_POOL_DEBUG, apr_palloc becomes apr_palloc_debug (include/apr_pools.h lines 451–454), which calls malloc() separately for every slice (pool_alloc() lines 1754–1791). When the pool is cleared, those slices are overwritten with 'A' and then free()d (line 1831, lines 1872–1882) — a trick to expose code that uses memory from a pool already cleared. The fact that the two slices in the example above came out 16 bytes apart tells us the APR in this container is not in debugging mode.

4. Clearing and destroying — clear, destroy, and the order of cleanups

apr_pool_clear()

This is the function that empties a pool, with the concurrency-check and Valgrind lines cut.

APR_DECLARE(void) apr_pool_clear(apr_pool_t *pool)
{
    apr_memnode_t *active;

    /* Run pre destroy cleanups */
    run_cleanups(&pool->pre_cleanups);
    /* ... */
    pool->pre_cleanups = NULL;
    /* ... */

    /* Destroy the subpools.  The subpools will detach themselves from
     * this pool thus this loop is safe and easy.
     */
    while (pool->child)
        apr_pool_destroy(pool->child);

    /* Run cleanups */
    run_cleanups(&pool->cleanups);
    /* ... */
    pool->cleanups = NULL;
    pool->free_cleanups = NULL;

    /* Free subprocesses */
    free_proc_chain(pool->subprocesses);
    pool->subprocesses = NULL;

    /* Clear the user data. */
    pool->user_data = NULL;

    /* Find the node attached to the pool structure, reset it, make
     * it the active node and free the rest of the nodes.
     */
    active = pool->active = pool->self;
    active->first_avail = pool->self_first_avail;
    /* ... */
    if (active->next == active) {
        /* ... */
        return;
    }

    *active->ref = NULL;
    allocator_free(pool->allocator, active->next);
    active->next = active;
    active->ref = &active->next;
    /* ... */
}

memory/unix/apr_pools.c lines 920–970

The order is the rule.

1. Run the pre-cleanups — tidy-ups registered with apr_pool_pre_cleanup_register() that must run "even before the child pools".
2. Destroy every child pool. Each child goes through apr_pool_destroy(), and steps 1–4 happen inside it in exactly the same way. In effect, grandchildren and below are tidied up from the bottom.
3. Run the cleanups.
4. Clean up subprocesses and drop user_data (dropping user_data happens only in clear).
5. Finally, rewind the memory. Put the first_avail of the first node (self, the one holding the pool struct) back where it started, and hand the remaining nodes back to the allocator, ring and all.

Note that step 5 has no free(). The first node is emptied by winding one pointer back, and the other nodes go to the allocator's lists. Just as allocating was pushing a pointer, freeing is rewinding one.

apr_pool_destroy() does steps 1–3 and the subprocess cleanup the same way (it doesn't reset user_data separately — that goes back along with the pool's memory; lines 972–1037); the only difference is that at the end it removes itself from the parent's child list and gives back even the first node. If that pool is the owner of its allocator, the allocator is destroyed too (lines 1033–1035). Only then are all the kept nodes free()d (apr_allocator_destroy() lines 198–217).

The four blue boxes are the same for clear and destroy; only the yellow boxes differ.

Why cleanups run "last registered first"

Here are the function that registers a cleanup and the one that runs them.

    if (p != NULL) {
        if (p->free_cleanups) {
            /* reuse a cleanup structure */
            c = p->free_cleanups;
            p->free_cleanups = c->next;
        } else {
            c = apr_palloc(p, sizeof(cleanup_t));
        }
        c->data = data;
        c->plain_cleanup_fn = plain_cleanup_fn;
        c->child_cleanup_fn = child_cleanup_fn;
        c->next = p->cleanups;
        p->cleanups = c;
    }

apr_pool_cleanup_register() — memory/unix/apr_pools.c lines 2514–2527

static void run_cleanups(cleanup_t **cref)
{
    cleanup_t *c = *cref;

    while (c) {
        *cref = c->next;
        (*c->plain_cleanup_fn)((void *)c->data);
        c = *cref;
    }
}

memory/unix/apr_pools.c lines 2660–2669

A new cleanup is inserted at the front of the list (c->next = p->cleanups; p->cleanups = c;), and running them walks from the front. So the last one registered runs first (LIFO, Last In First Out). The header states it plainly: "Cleanups are performed in the reverse order they were registered" (include/apr_pools.h lines 608–609).

Why reverse? Because things made later are usually built on top of things made earlier. If you open a file and then build a buffer on that file, the buffer must be flushed before the file is closed. Tidying up in the reverse of creation order keeps those dependencies intact automatically.

(*c->plain_cleanup_fn)((void *)c->data) is C syntax for calling a function through a function pointer. It gets back exactly the data that was passed in at registration. Notice also that the cleanup_t struct itself is taken with apr_palloc() from that same pool — the cleanup list disappears along with the pool, too.

The fourth argument of the register function, child_cleanup_fn, is the one called in a child process made with fork() just before it execs another program. When APR launches a process it calls apr_pool_cleanup_for_exec() (threadproc/unix/proc.c line 425), which runs child_cleanup_fn for every pool (lines 2673–2695). If there's nothing to do, you pass apr_pool_cleanup_null, which does nothing.

APR uses this mechanism itself. For example, apr_socket_accept() registers socket_cleanup on the accepted socket (network_io/unix/sockets.c lines 383–384), so the socket is closed when the pool it was made in is cleared. apr_file_open() registers a cleanup on files the same way (file_io/unix/open.c lines 245–249). This is why a pool ties together the lifetime of all resources, not just memory.

Example 2: watching the order

Part 2 of pooldemo.c. It registers A, B and C on one pool in that order, S on a child pool, and P as a pre-cleanup — then clears the pool.

/* An example written for this post (not httpd source) */
static apr_status_t say(void *data)
{
    printf("  cleanup: %s\n", (const char *)data);
    return APR_SUCCESS;
}
    /* ... inside main() ... */
    /* 2. The order cleanups run in */
    printf("[2] register A, B, C on pool; S on a subpool; P as a pre-cleanup\n");
    apr_pool_cleanup_register(pool, "A", say, apr_pool_cleanup_null);
    apr_pool_cleanup_register(pool, "B", say, apr_pool_cleanup_null);
    apr_pool_cleanup_register(pool, "C", say, apr_pool_cleanup_null);
    apr_pool_create(&sub, pool);
    apr_pool_cleanup_register(sub, "S (subpool)", say, apr_pool_cleanup_null);
    apr_pool_pre_cleanup_register(pool, "P (pre-cleanup)", say);
    printf("  apr_pool_clear(pool):\n");
    apr_pool_clear(pool);

    /* After a clear, the same memory is reused from the start */
    first = apr_palloc(pool, 13);
    printf("  first palloc after clear returns the old address: %s\n",
           first == a ? "yes" : "no");

[2] register A, B, C on pool; S on a subpool; P as a pre-cleanup
  apr_pool_clear(pool):
  cleanup: P (pre-cleanup)
  cleanup: S (subpool)
  cleanup: C
  cleanup: B
  cleanup: A
  first palloc after clear returns the old address: yes

Pre-cleanup → child pool → cleanups, last registered first. Exactly the order of apr_pool_clear() we just read. The last line says the first memory taken after the clear is at the same address a that the very first allocation got before the clear — because the first node's first_avail was wound back to its start.

Easy to get confused

• Cleanups follow "reverse registration order", not only "the pool tree". In the output above, the child pool's S ran before the parent's A, B and C. By the time the parent's cleanups run, the child pool is already gone — so a parent pool's cleanup must not touch a child pool's memory.
• A cleanup may use memory from its own pool. Step 5, the rewind, happens after all cleanups have run. The header also says it's safe for a cleanup to allocate from the pool being cleaned up — and warns that child pools created inside a cleanup will not be cleaned up automatically (include/apr_pools.h lines 609–614).
• To remove a cleanup early, use apr_pool_cleanup_kill(). If you closed a file early, you remove its registration so it isn't closed again when the pool is cleared. It finds the entry whose data and function both match (lines 2566–2627).

5. httpd's pool tree — who creates what, and when it's thrown away

Now over to httpd. httpd's pools are layered by lifetime.

Event MPM. The further down, the shorter the lifetime. Arrows go parent → child.

Process and configuration — process, pconf, ptemp, plog

The first thing main() calls, init_process(), creates a pool with no parent and tags it "process" (server/main.c line 352, line 375). When the parent is NULL, APR uses apr_global_pool, which it created first thing, as the parent (apr_pools.c lines 1049–1050; the tag is on line 680). This pool is destroyed once, when httpd exits, and every pool below it goes with it (server/main.c line 302).

Below it, the pool that holds the configuration is pconf.

static void reset_process_pconf(process_rec *process)
{
    if (process->pconf) {
        apr_pool_clear(process->pconf);
        ap_server_conf = NULL;
    }
    else {
        apr_pool_create(&process->pconf, process->pool);
        apr_pool_tag(process->pconf, "pconf");
    }
    apr_pool_pre_cleanup_register(process->pconf, NULL, deregister_all_hooks);
}

server/main.c lines 329–340

The first time it creates the pool; after that it clears it. Every time you restart with apachectl restart or graceful, the do { … } while loop in main() calls this function (at first startup, too, once in the loop's first pass; lines 811–812), throwing away the memory that held the previous configuration in one go before reading the config again. Cleanups that modules registered on pconf run at this point too. deregister_all_hooks, registered as a pre-cleanup, removes every registered hook — because when the config is read again, the hooks are registered again. Why the config is read again at all is covered in Part 5.

main() has two more pools. The comments on the variable declarations tell you what they're for.

    apr_pool_t *pconf;
    apr_pool_t *plog; /* Pool of log streams, reset _after_ each read of conf */
    apr_pool_t *ptemp; /* Pool for temporary config stuff, reset often */

server/main.c lines 537–539

• ptemp is created as a child of pconf (lines 707–708), used while reading the config file, and destroyed as soon as the post_config phase is over (line 808, line 876). It keeps intermediate results from parsing the config file from hanging around for as long as the server runs.
• plog is a child of the process pool (lines 705–706) and is cleared right before each open_logs phase, which opens the log files (line 794, line 863).

This is why hooks like pre_config and post_config take the three pools (pconf, plog, ptemp) as arguments (line 742, line 802). A module picks a pool by asking "how long does this data need to live?"

Connections — ptrans and c->pool

The event MPM's listener thread prepares a connection pool, ptrans, for every connection it accepts. It takes one from a recycle list, and creates one if the list is empty.

                    if (ptrans == NULL) {
                        /* create a new transaction pool for each accepted socket */
                        apr_allocator_t *allocator = NULL;

                        rc = apr_allocator_create(&allocator);
                        if (rc == APR_SUCCESS) {
                            apr_allocator_max_free_set(allocator,
                                                       ap_max_mem_free);
                            rc = apr_pool_create_ex(&ptrans, pconf, NULL,
                                                    allocator);
                            if (rc == APR_SUCCESS) {
                                apr_pool_tag(ptrans, "transaction");
                                apr_allocator_owner_set(allocator, ptrans);
                            }
                        }

server/mpm/event/event.c lines 2242–2256

Each connection pool gets its own allocator, with the MaxMemFree value from section 2, and the pool is made the allocator's owner. A connection is handled by several worker threads over its life, but since each connection has its own allocator for swapping nodes, it never fights other connections for that lock.

When the connection ends, ptrans is not destroyed but cleared and put on the recycle list. It is destroyed only when the list is already full.

    if (queue_info->max_recycled_pools >= 0) {
        apr_uint32_t n = apr_atomic_read32(&queue_info->recycled_pools_count);
        if (n >= queue_info->max_recycled_pools) {
            apr_pool_destroy(pool_to_recycle);
            return;
        }
        apr_atomic_inc32(&queue_info->recycled_pools_count);
    }

    apr_pool_clear(pool_to_recycle);

ap_queue_info_push_pool() — server/mpm_fdqueue.c lines 227–236

The event MPM's close_connection() calls this function after closing the socket (event.c lines 924–931). The size of the recycle list is set to 3/4 of ThreadsPerChild unless MaxMemFree is unlimited (lines 2684–2690).

The prefork MPM is simpler. Each child process handles one connection at a time, so it creates ptrans once under the child pool pchild (server/mpm/prefork/prefork.c lines 469–470) and reuses it, emptying it with apr_pool_clear(ptrans) before accepting each connection (line 580). MaxMemFree goes into the child process's allocator at lines 444–447. How each MPM shares out connections is covered in Part 8.

With ptrans in hand, the core builds the conn_rec. But in trunk, c->pool is not ptrans itself.

    conn_rec *c = (conn_rec *) apr_pcalloc(ptrans, sizeof(conn_rec));
    /* ... */
    apr_pool_create(&pool, ptrans);
    apr_pool_tag(pool, "master_conn");
    c->pool = pool;

server/core.c lines 5516–5527

The conn_rec struct goes in ptrans, and the connection's data goes in its child pool, master_conn. On the 2.4.x branch the same function uses ptrans directly with c->pool = ptrans;. Trunk's extra layer looks like it exists for putting several "secondary connections" on top of one connection ("secondary_conn" near the same function, lines 5610–5611), but I didn't confirm that intent from the source. Either way, c->pool is tidied up whenever ptrans is cleared — in 2.4.x because it's the same pool, in trunk because it's a child pool.

Requests — when does r->pool go away?

When a request comes in, ap_create_request() creates a request pool as a child of c->pool.

AP_DECLARE(request_rec *) ap_create_request(conn_rec *conn)
{
    request_rec *r;
    apr_pool_t *p;

    apr_pool_create(&p, conn->pool);
    apr_pool_tag(p, "request");
    r = apr_pcalloc(p, sizeof(request_rec));
    AP_READ_REQUEST_ENTRY((intptr_t)r, (uintptr_t)conn);
    r->pool            = p;
    r->connection      = conn;

server/protocol.c lines 1058–1068

The request_rec struct itself lives inside the request pool. When the request pool goes, the memory r points to goes with it.

So when does the request pool go? Not "when the handler has finished building the response" — the response may not have been fully sent to the socket yet. httpd appends a special bucket called EOR (End Of Request) after the response data and sends it down the output filters (modules/http/http_request.c lines 343–362). Buckets and output filters are covered in detail in Part 11; for now, think of it as "the end marker of the queue of response data flowing out". When that marker has been fully processed and destroyed, the request pool is destroyed.

static void eor_bucket_destroy(void *data)
{
    ap_bucket_eor *h = data;

    if (apr_bucket_shared_destroy(h)) {
        request_rec *r = h->r;
        if (r) {
            /* eor_bucket_cleanup will be called when the pool gets destroyed */
            apr_pool_destroy(r->pool);
        }
        apr_bucket_free(h);
    }
}

server/eor_bucket.c lines 100–112

The eor_bucket_cleanup the comment mentions was registered on the request pool as a pre-cleanup when the EOR bucket was created (line 88), and inside it calls the log_transaction hook, which writes the access log (line 40). Why a pre-cleanup rather than a normal cleanup is in a comment as well: "a module may create a sub-pool which is still needed during the log_transaction hook" (lines 85–86). In section 4's order, only pre-cleanups run before the child pools are destroyed.

Easy to get confused

• "The request is over" is not the moment the handler returns. The request pool is destroyed when the EOR bucket has passed all the output filters and is destroyed. If the output filters can't write the whole response to the socket and set the rest aside, the EOR bucket stays behind with it and is destroyed later. The comment in http_request.c also says the EOR bucket has either been handled by an output filter (deleted, or moved to a buffered_bb) or an error occurred before that (lines 365–370).
• Subrequests are children of the request pool. A subrequest made by functions like ap_sub_req_lookup_uri() gets a "subrequest" pool under r->pool (server/request.c lines 2056–2057), and ap_destroy_sub_req() can destroy just that pool (lines 2596–2600). If you forget, it's still tidied up when the parent request ends. Subrequests are covered in Part 10.

6. Example 3: one keep-alive connection, two requests

Let's check section 5 for ourselves. This module registers two cleanups on the request pool every time its handler runs, and one cleanup on the connection pool once per connection. When a cleanup runs, it writes a line to the error log. It also writes a line from the log_transaction hook, which writes the access log, to see which comes first: that or the request pool's cleanups.

/* An example written for this post (not httpd source) */
#include "httpd.h"
#include "http_config.h"
#include "http_protocol.h"
#include "http_log.h"
#include "apr_strings.h"

APLOG_USE_MODULE(poolwatch);

/* Called when the pool is tidied up. data is the string passed at registration. */
static apr_status_t say_bye(void *data)
{
    ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, NULL, "cleanup: %s",
                 (const char *)data);
    return APR_SUCCESS;
}

static int poolwatch_handler(request_rec *r)
{
    conn_rec *c = r->connection;

    if (strcmp(r->handler, "poolwatch")) {
        return DECLINED;
    }

    /* Register on the connection pool only once per connection (leave a mark in c->notes) */
    if (!apr_table_get(c->notes, "poolwatch")) {
        apr_table_setn(c->notes, "poolwatch", "1");
        apr_pool_cleanup_register(c->pool,
                                  apr_psprintf(c->pool, "c->pool (connection %ld)", c->id),
                                  say_bye, apr_pool_cleanup_null);
    }

    /* Register two on the request pool per request — to compare registration and run order */
    apr_pool_cleanup_register(r->pool,
                              apr_pstrcat(r->pool, "r->pool ", r->uri, " #1 (registered first)", NULL),
                              say_bye, apr_pool_cleanup_null);
    apr_pool_cleanup_register(r->pool,
                              apr_pstrcat(r->pool, "r->pool ", r->uri, " #2 (registered second)", NULL),
                              say_bye, apr_pool_cleanup_null);

    ap_log_rerror(APLOG_MARK, APLOG_NOTICE, 0, r,
                  "handler: %s (r->pool's parent is c->pool: %s)", r->uri,
                  apr_pool_parent_get(r->pool) == c->pool ? "yes" : "no");

    ap_set_content_type(r, "text/plain");
    ap_rprintf(r, "hello from %s\n", r->uri);
    return OK;
}

/* The access-logging phase — to see whether it comes before or after the request pool's cleanups */
static int poolwatch_log(request_rec *r)
{
    ap_log_rerror(APLOG_MARK, APLOG_NOTICE, 0, r, "log_transaction: %s", r->uri);
    return DECLINED;
}

static void register_hooks(apr_pool_t *p)
{
    ap_hook_handler(poolwatch_handler, NULL, NULL, APR_HOOK_MIDDLE);
    ap_hook_log_transaction(poolwatch_log, NULL, NULL, APR_HOOK_MIDDLE);
}

/* APLOG_USE_MODULE is already at the top, so define the module like this instead of AP_DECLARE_MODULE */
module AP_MODULE_DECLARE_DATA poolwatch_module = {
    STANDARD20_MODULE_STUFF, NULL, NULL, NULL, NULL, NULL, register_hooks
};

Notice that the strings passed to the cleanups are built in that same pool (apr_pstrcat(r->pool, …)). As section 4 showed, the pool's memory is still alive when its cleanups run, so this is safe. apr_pstrcat(), which joins strings, and apr_psprintf(), which builds a string like printf, both take their memory from a pool too.

Compile it with apxs -c, start httpd with a config that puts SetHandler poolwatch on <Location /pw>, and request two URLs one after the other with a single curl. For the second request to the same server, curl reuses the first connection (keep-alive). The full config and commands are in run.sh in the example folder; the error-log format is trimmed with ErrorLogFormat "[%l] %M".

curl -sv http://127.0.0.1:8080/pw/a http://127.0.0.1:8080/pw/b

* Connected to 127.0.0.1 (127.0.0.1) port 8080
* Connection #0 to host 127.0.0.1 left intact
hello from /pw/a
* Re-using existing connection with host 127.0.0.1
* Connection #0 to host 127.0.0.1 left intact
hello from /pw/b

The error log, read after curl has exited but before stopping httpd:

[notice] AH00489: Apache/2.5.1-dev (Unix) configured -- resuming normal operations
[notice] AH00094: Command line: 'httpd -f /tmp/u04.conf'
[notice] handler: /pw/a (r->pool's parent is c->pool: yes)
[notice] log_transaction: /pw/a
[notice] cleanup: r->pool /pw/a #2 (registered second)
[notice] cleanup: r->pool /pw/a #1 (registered first)
[notice] handler: /pw/b (r->pool's parent is c->pool: yes)
[notice] log_transaction: /pw/b
[notice] cleanup: r->pool /pw/b #2 (registered second)
[notice] cleanup: r->pool /pw/b #1 (registered first)
[notice] cleanup: c->pool (connection 128)

(The number in connection 128 is computed from the slot of the child process and thread that handled the connection, so it can differ from run to run. When I ran it again, I got connection 64.)

Yellow is where pools are created, blue is where tidying up starts, green is where cleanups actually run.

Four things can be read from the log.

1. r->pool's parent is c->pool (yes) — exactly ap_create_request() from section 5.
2. Each request gets its own request pool, created and destroyed separately. The cleanups for /pw/a were done before the handler for /pw/b was even called. Even on the same connection, each request's memory went back as soon as that request ended.
3. log_transaction comes before the request pool's cleanups, because eor_bucket_cleanup is a pre-cleanup. And the request pool's cleanups run #2 → #1, the reverse of registration.
4. The connection pool's cleanup runs once, after both requests, when the connection is closed. It was already in the log before httpd was stopped, so it ran because the connection closed, not because the server shut down.

I did get stuck once while building this example. At first I used both APLOG_USE_MODULE(poolwatch); at the top of the file and AP_DECLARE_MODULE(poolwatch) at the bottom, and got a compile error saying aplog_module_index was defined twice. That's because AP_DECLARE_MODULE is a macro that calls APLOG_USE_MODULE again internally (include/http_config.h lines 456–469). To use the log functions above the module definition, put APLOG_USE_MODULE at the top as shown and define the module directly at the bottom with module AP_MODULE_DECLARE_DATA …. The module struct is covered in Part 6.

7. The two classic mistakes

Pools let you forget "when do I throw this away", but in exchange you must not get "which pool do I take it from" wrong. There are two ways to get it wrong.

Mistake 1: taking from a pool that lives too long — a leak

If you take per-request data from a long-lived pool like c->pool or pconf, that memory doesn't come back until that pool is cleared. And since pools have no function for returning a single piece, there's no way to give it back in the meantime. Send thousands of requests over one keep-alive connection and c->pool grows with every request; if you took it from pconf, it grows until the next restart.

Part 3 of pooldemo.c shows this. It takes 1,000 bytes 10,000 times — once from a long-lived pool, and once from a child pool cleared after each "request's worth".

/* An example written for this post (not httpd source) */
    /* 3. Allocating per request from a long-lived pool */
    apr_pool_create(&longlived, NULL);
    apr_pool_create(&perreq, longlived);
    before = rss_kb();
    for (i = 0; i < 10000; i++) {
        memset(apr_palloc(longlived, 1000), 'x', 1000);  /* mistake: from the long-lived pool */
    }
    printf("[3] 10000 x palloc(1000) from a long-lived pool: RSS +%ld KB\n",
           rss_kb() - before);

    before = rss_kb();
    for (i = 0; i < 10000; i++) {
        memset(apr_palloc(perreq, 1000), 'x', 1000);     /* one request's worth */
        apr_pool_clear(perreq);               /* clear when the request ends */
    }
    printf("    same, but from a subpool cleared each time: RSS +%ld KB\n",
           rss_kb() - before);

rss_kb() is a function written for the example that reads /proc/self/statm and returns, in KB, the memory this process is actually using (RSS).

[3] 10000 x palloc(1000) from a long-lived pool: RSS +10664 KB
    same, but from a subpool cleared each time: RSS +0 KB

(The number on the first line varies a little from run to run; three runs gave 10664, 10772 and 10732.)

The long-lived pool holds on to all 10,000 × 1,000 bytes ≈ 10MB, while the child pool cleared each time keeps rewinding the same node and doesn't grow.

Inside httpd the fix is the same. If you have to repeat work tied to a long-lived pool, create a child pool, work inside it, and clear or destroy it each round. httpd's core does exactly that. ap_directory_walk(), which finds directory configuration, creates a temporary pool "directory_walk_rxpool" under r->pool when matching paths against regular expressions, and destroys it when done (server/request.c lines 1351–1352, line 1412). Using ptemp only while reading the config (section 5) is the same idea.

Mistake 2: taking from a pool that dies too soon — using memory that's gone

The other direction is more dangerous. If you store an address from the request pool somewhere that outlives the request — a struct in the connection pool, a module's global variable, another thread — then the moment the request ends, that address points at memory already given back. The next request reuses that node, so the text may have changed, or something entirely different may be there. The last line of section 4's example — the first address after a clear is the same as an old one — is exactly that reuse. Because it doesn't fail right away but only sometimes, and somewhere unrelated, it's hard to track down.

httpd's core nails this boundary down in a comment too, right after sending section 5's EOR bucket to the output filters.

    apr_brigade_cleanup(bb);

    /* From here onward, it is no longer safe to reference r
     * or r->pool, because r->pool may have been destroyed
     * already by the EOR bucket's cleanup function.

modules/http/http_request.c lines 371–375

"From here on, don't touch r or r->pool." The function is called ap_process_request_after_handler(), and even inside it, past a certain point, it can't touch the very request it was handling.

The APR header states the general rule. When pool B is a child of pool A, putting A's data into a table made in B is safe, but putting B's data into a table made in A is not — "B can be cleared/destroyed before A is (which would leave dangling pointers …)" (include/apr_pools.h lines 730–737). In one line: whatever holds a pointer must not outlive what it points to.

The numbers for mistake 1 are the real output of part 3 of pooldemo above (the first line varies a little from run to run).

Easy to get confused

• Copy what must outlive the request. If you need to remember a value from a request at the connection level, make a copy in the longer-lived pool, e.g. apr_pstrdup(c->pool, value). But do that on every request and you're back to mistake 1. Think about whether it's needed only once per connection.
• Putting something from ptemp into a config struct in pconf is the same mistake. It works fine while the config is being read, then breaks the moment ptemp is destroyed after post_config.

8. Compared with Java's GC

If you've been following the Spring series, you'll remember Java never made us worry about this. In Java, the garbage collector (GC) finds objects "nobody points to any more" and reclaims them for us. The Java column in the table below describes typical JVM behaviour; it isn't something I checked against source for this post.

The strength of pools is that they're predictable. A request pool is always tidied up when the request ends, and the cost is just putting a few nodes back. The price is that the programmer must know lifetimes exactly. Both mistakes in section 7 are getting "how long does this data need to live" wrong. GC makes that call for you, and in return leaves when and how much it works up to the runtime.

The table's "When memory is thrown away" and "Who decides lifetime" rows laid out in time order. The bottom row (Java) is typical JVM behaviour and was not checked against source in this post.

The reverse order of cleanups has a counterpart in Spring. In Spring Part 6 we saw that when a Spring container closes, it destroys singleton beans in reverse registration order, and before destroying a bean it first destroys the beans that depend on it. Same reason: things made later lean on things made earlier. And httpd's restart — clear pconf and read the config again — is a lot like closing one container and opening a fresh one.

What this post leaves out

To keep to the main thread, the following are only named, with their locations.

• Attaching data to a pool by key, apr_pool_userdata_set()/get() — modules use this to remember values across restarts (apr_pools.c from line 2430).
• Parentless pools, apr_pool_create_unmanaged_ex() — pools that don't hang on the tree and have their own allocator (lines 1138–1198).
• How apr_psprintf() works — since it doesn't know the output length in advance, it has a special path that writes straight into the current node and moves to a new node if it runs out (the comment at lines 1204–1216 and what follows).
• Subprocess cleanup, free_proc_chain() — the part that ends child processes registered on a pool with signals (step 4 of clear/destroy).
• Pool-debugging mode as a whole — the per-level features of APR_POOL_DEBUG (owner-thread checks, lifetime checks, finding a pool with apr_pool_find()) are described at include/apr_pools.h lines 95–131 and implemented in apr_pools.c below #else /* APR_POOL_DEBUG */ (from line 1453).
• The event MPM's pskip and pruntime pools — a long comment on where to put things that must outlive the connection pools is at event.c lines 2647–2673.
• Memory for buckets, apr_bucket_alloc — buckets get their memory from an allocator separately rather than from a pool. Covered in Part 11.

Summary

What's next

Now that we've seen how pools tie lifetimes together, we go to where the root of that tree is planted. The do { … } while loop in main() — which this post only described as "clearing pconf on every restart" — and the reason httpd reads its config file twice continue in Part 5: main() and Startup — Why the Config Is Read Twice, and Restarts. We'll count for ourselves how many times pre_config and post_config are called.

• Part 1: How I Read It with AI, and the Map
• Part 2: Before Reading a Server Written in C — Structs, Function Pointers, Macros
• Part 3: The Big Picture — Core, APR, MPM, Modules and the Three Big Structs
• From the Spring series, on lifetimes and tidy-up order: Spring Part 6: Lifecycle Callbacks and PostProcessors
• The same idea built differently: ngx_pool_t in nginx Part 3, MEM_ROOT in MariaDB Part 2