Digging into Apache httpd's Source 8: MPMs — How prefork, worker and event Share Out Connections

What this post covers

This is the eighth post in the "Digging into Apache httpd's Source" series. Part 5 followed main() up to the point where, after reading the configuration, it calls ap_run_mpm(), and inside that the MPM starts child processes and waits for signals. This post opens up the inside of ap_run_mpm(): the component that accepts connections and decides who handles them.

There are four questions to answer.

• What is an MPM, and why can it be swapped with a single LoadModule line?
• How do prefork, worker and event differ in the way they accept and handle a connection — and in particular, what does event do with idle keep-alive connections?
• When does the parent process add or remove children, and what is the scoreboard it bases that decision on?
• Which variables in the code do directives like StartServers, MaxRequestWorkers and ThreadsPerChild turn into?

Every line number and every detail in this post refers to that commit. Part 1 explains how to get the source at the same commit and set up the example environment. Each MPM file is 1,600 to 4,500 lines long, and most of that is signal handling, restarts and error handling. This post keeps only the main thread — "where does a connection go?" — that someone opening the source for the first time can follow; what was left out is named at the end, under "What this post leaves out".

Words to know first

Processes and threads: a process is one running program with its own memory. It can copy itself with fork() to create a child process, and after the copy the two cannot see each other's memory. A thread is a flow of execution running inside a process at the same time as others, sharing the same memory. Processes are isolated — one dying leaves the others untouched — but heavy; threads are light, but one thread corrupting memory takes the whole process down. httpd's "one parent process + several children" layout was covered in Part 5.

MPM (Multi-Processing Module): the module that accepts connections on the listening sockets and decides which process or thread handles each one. Actually reading the request and answering it is the job of the core and other modules; the MPM only decides "who takes it" and "how many workers to keep".

Listening socket and accept: Listen 80 creates a socket opened on port 80 that says "I take connections here". When a client connects, the operating system lines it up on that socket, and when the server calls accept() it hands back a new socket (the connection socket) just for that client. The rest of the conversation happens on this new socket.

Keep-alive: in HTTP/1.1 the connection can stay open after a response, and the next request can arrive on the same connection. The server doesn't know when that next request will come, so it waits for KeepAliveTimeout (5 seconds by default). "Who is holding the connection while we wait?" is the central question of this post.

poll / epoll: an operating-system facility where you register many sockets at once and ask "tell me which of these have something to read". On Linux it's epoll; APR (the operating-system abstraction layer from Part 3) wraps it in the apr_pollset_* functions. It lets one thread watch thousands of sockets.

Hooks: the core defines places where modules can hang functions, and the core calls them at that point. Hooks are run either RUN_FIRST ("stop at the first one that handles it") or RUN_ALL ("call them all"). Details are in Part 6.

1. Why the MPM is a swappable part

The right answer depends on the OS and the workload

There's no single right way to share out connections. Some operating systems had no threads, or unreliable ones; modules that use thread-unsafe libraries (the old PHP module is the classic example) are only safe with one request per process; Windows has no fork() and needs a completely different approach. At this commit the server/mpm/ folder in fact holds eight MPMs — prefork, worker, event, winnt (Windows), mpmt_os2 (OS/2), netware, and the trunk-only experimental motorz and simple. The 2.4.x branch has no motorz or simple folders.

So httpd doesn't bake this decision into the core; it pulls it out into one module. This line in httpd.conf is it:

LoadModule mpm_event_module modules/mod_mpm_event.so

Change it to mod_mpm_prefork.so and the very same httpd binary accepts connections in a completely different way. (This example environment was built with --enable-mpms-shared=all, so each MPM is a separate .so. You can also build one MPM straight into the binary.)

The contract between core and MPM: one mpm hook

What the core expects from an MPM boils down to one hook declaration in include/ap_mpm.h.

AP_DECLARE_HOOK(int, mpm, (apr_pool_t *pconf, apr_pool_t *plog, server_rec *server_conf))

include/ap_mpm.h line 93

The comment right above it says the MPM "is responsible for controlling the parent and child processes" and "will run until a restart/shutdown is indicated" (lines 83–92), and the long comment at the top of the file promises that when a connection arrives the MPM calls into the core through ap_process_connection() (lines 37–81). The hook is implemented as RUN_FIRST (server/mpm_common.c lines 103–105), so once the one hooked function takes the job, that's it. All main() does is call it after reading the config (server/main.c line 882).

Each MPM hooks its functions in its register_hooks function, exactly like the ordinary modules in Part 6. Here is prefork:

    ap_hook_pre_config(prefork_pre_config, NULL, NULL, APR_HOOK_REALLY_FIRST);
    ap_hook_check_config(prefork_check_config, NULL, NULL, APR_HOOK_MIDDLE);
    ap_hook_mpm(prefork_run, NULL, NULL, APR_HOOK_MIDDLE);
    ap_hook_mpm_query(prefork_query, NULL, NULL, APR_HOOK_MIDDLE);
    ap_hook_mpm_get_name(prefork_get_name, NULL, NULL, APR_HOOK_MIDDLE);

server/mpm/prefork/prefork.c lines 1517–1521

worker hooks worker_run (worker.c line 2402) and event hooks event_run (event.c line 4381) in the same place. Functions like prefork_run are all static. In C a static function can't be called by name from outside its file (Part 2), so the core doesn't know a single MPM function name. It calls them only through the function pointers hanging on the hooks. That is what "swappable" really means.

Exactly one — ap_check_mpm()

There must be exactly one MPM. ap_read_config() checks with this function after it has processed the file (server/config.c line 2355).

AP_DECLARE(const char *)ap_check_mpm(void)
{
    static const char *last_mpm_name = NULL;

    if (!_hooks.link_mpm || _hooks.link_mpm->nelts == 0)
        return "No MPM loaded.";
    else if (_hooks.link_mpm->nelts > 1)
        return "More than one MPM loaded.";

    if (last_mpm_name) {
        if (strcmp(last_mpm_name, ap_show_mpm())) {
            return "The MPM cannot be changed during restart.";
        }
    }
    else {
        last_mpm_name = apr_pstrdup(ap_pglobal, ap_show_mpm());
    }

    return NULL;
}

server/mpm_common.c lines 656–675

_hooks.link_mpm is the array of functions hooked onto the mpm hook, and nelts is how many there are. Zero gives "No MPM loaded.", two or more gives "More than one MPM loaded.". The static local variable last_mpm_name keeps its value after the function returns, so it remembers the MPM name from the first read and refuses a switch to a different MPM on restart (Part 5).

An MPM attaches to the core through two hooks (run and query). Blue is procedure, red is an error, grey is the side that asks the MPM.

Other modules ask the MPM — ap_mpm_query()

If the MPM can be swapped, other modules need a way to find out "does the current MPM use threads? how many processes at most?". That's ap_mpm_query(). The questions are numbered by the AP_MPMQ_* constants (include/ap_mpm.h lines 149–186), and the function itself just runs the mpm_query hook.

AP_DECLARE(apr_status_t) ap_mpm_query(int query_code, int *result)
{
    apr_status_t rv;

    if (ap_run_mpm_query(query_code, result, &rv) == DECLINED) {
        rv = APR_EGENERAL;
    }

    return rv;
}

server/mpm_common.c lines 496–505

The answers come from a switch in each MPM. prefork answers "threaded?" with AP_MPMQ_NOT_SUPPORTED and "threads per child" with 1 (prefork.c lines 316–367); event answers "is this an async MPM?" (AP_MPMQ_IS_ASYNC) with 1 (event.c lines 697–699).

A real user of that answer is the http module, which handles HTTP. At startup it asks whether the MPM is async (modules/http/http_core.c lines 346–348), and when handling a connection it takes one of two paths.

static int ap_process_http_connection(conn_rec *c)
{
    if (async_mpm && !c->clogging_input_filters) {
        return ap_process_http_async_connection(c);
    }
    else {
        return ap_process_http_sync_connection(c);
    }
}

modules/http/http_core.c lines 247–255

The sync path keeps reading the connection's requests in a while loop; the async path handles one request and returns. The insides of those two functions are in Part 9. This post looks, in section 4, at what that difference looks like on the MPM side.

Example 1: no MPM, two MPMs, and httpd -V

The examples in this post were run in a Docker container (Ubuntu 24.04) with httpd 2.5.1-dev built unchanged from the reference commit (e4af30fe6d). The example folder is ex/u08-mpm/, and a single run.sh runs all of this post's examples. The config files differ only in the LoadModule line for the MPM; everything else (Listen 8080, mod_status, LogLevel debug) is the same. The commands below are a shortened excerpt of run.sh.

# An example written for this post (not part of httpd's source)
httpd -t -f /tmp/none.conf     # a config that loads no MPM
httpd -t -f /tmp/two.conf      # a config that loads both mod_mpm_event and mod_mpm_worker
for m in prefork worker event; do
  httpd -V -f /tmp/$m.conf | grep -A2 "Server MPM"
done

=== 1. exactly one MPM ===
  AH00534: httpd: Configuration error: No MPM loaded.
  AH00534: httpd: Configuration error: More than one MPM loaded.

=== 2. same httpd, different MPM: the MPM lines of httpd -V (ap_mpm_query answers) ===
--- prefork
  Server MPM:     prefork
    threaded:     no
      forked:     yes (variable process count)
--- worker
  Server MPM:     worker
    threaded:     yes (fixed thread count)
      forked:     yes (variable process count)
--- event
  Server MPM:     event
    threaded:     yes (fixed thread count)
      forked:     yes (variable process count)

The first two lines are the strings ap_check_mpm() returned, as is. The threaded: and forked: lines below come from show_mpm_settings(), which handles httpd -V: it asks ap_mpm_query(AP_MPMQ_IS_THREADED, …) and AP_MPMQ_IS_FORKED and turns the answers into text (server/main.c lines 64–100). Same binary, different answers.

Easy to get wrong: Listen is an MPM directive

Listen 80 looks like a core setting, but in the source the directive is registered by the MPM. include/ap_listen.h has a LISTEN_COMMANDS macro bundling Listen, ListenBacklog, SendBufferSize and others (lines 176–190), and every MPM's directive table starts with it (e.g. prefork.c line 1584, event.c line 4510). ap_setup_listeners(), which opens the listening sockets, is also called from each MPM's open_logs hook (prefork.c line 1315). So you might expect none.conf in Example 1, with no MPM loaded, to fail with "invalid command 'Listen'" — but httpd -t actually stopped at "No MPM loaded.". That's because ap_check_mpm() runs right after the config file has been read into a tree and LoadModule has been processed, before the remaining directives are executed (server/config.c lines 2348–2362). The two-stage processing of the config is covered in Part 7.

2. prefork — one connection per process

prefork is the oldest and simplest approach. As the name says, child processes are forked in advance (pre-fork) before connections arrive, and each one accepts and handles connections one at a time with its single thread.

The red box is prefork's weak spot: one connection takes up a whole child process.

A day in a child's life: the child_main() loop

A child is born by fork() in make_child() (prefork.c line 745), goes straight into child_main(), and runs this loop for the rest of its life. (/* ... */ marks places shortened for this post.)

        apr_pool_clear(ptrans);

        if ((ap_max_requests_per_child > 0
             && requests_this_child++ >= ap_max_requests_per_child)) {
            clean_child_exit(0);
        }

        (void) ap_update_child_status(sbh, SERVER_READY, (request_rec *) NULL);

        /*
         * Wait for an acceptable connection to arrive.
         */

        /* Lock around "accept", if necessary */
        SAFE_ACCEPT(accept_mutex_on());
        /* ... with several Listens, poll to pick a listening socket that is readable ... */
        status = lr->accept_func(&csd, lr, ptrans);

        SAFE_ACCEPT(accept_mutex_off());      /* unlock after "accept" */
        /* ... error handling ... */
        current_conn = ap_run_create_connection(ptrans, ap_server_conf, csd, my_child_num, sbh, bucket_alloc);
        if (current_conn) {
#if APR_HAS_THREADS
            current_conn->current_thread = thd;
#endif
            ap_process_connection(current_conn, csd);
            ap_lingering_close(current_conn);
        }

server/mpm/prefork/prefork.c lines 580–695

1. Clear the pool. ptrans is the memory pool used for one connection; it is cleared and reused before each accept (Part 4).
2. Check whether it's time to retire. If it has handled more connections than MaxConnectionsPerChild, the child exits on its own. That directive is registered by the core, not the MPM (server/core.c line 4994), and lands in the shared variable ap_max_requests_per_child (server/mpm_common.c line 411).
3. Write "idle" (SERVER_READY) on the scoreboard. The scoreboard is section 6.
4. Accept. lr->accept_func is a function pointer stored in the listening-socket struct ap_listen_rec (include/ap_listen.h line 40, lines 55–87). On Unix it is set to ap_unixd_accept() (prefork.c line 559), which calls APR's apr_socket_accept() (os/unix/unixd.c line 303). Function pointers are explained in Part 2.
5. Handle the connection. ap_process_connection() is the core's entry point for reading this connection's requests and answering them (Part 9). Until it returns — including the time spent waiting for the next keep-alive request — this child process can do nothing else.

The accept mutex and SAFE_ACCEPT

What happens when several children call accept() on the same listening socket at once? With one Listen and an operating system that handles it well, nothing bad. But with several Listens, the children first have to poll to see "which socket has a client" and then accept, and if another child grabs that client in between, a child can end up waiting on the wrong socket. So prefork lets only the child holding an inter-process lock (the accept mutex) accept. Whether to lock at all is decided by one macro.

#ifdef SINGLE_LISTEN_UNSERIALIZED_ACCEPT
#define SAFE_ACCEPT(stmt) (ap_listeners->next ? (stmt) : APR_SUCCESS)
#else
#define SAFE_ACCEPT(stmt) (stmt)
#endif

server/mpm/prefork/prefork.c lines 310–314

SINGLE_LISTEN_UNSERIALIZED_ACCEPT marks "an OS where it's safe not to lock when there is a single listening socket", and configure turns it on for Linux (2.2 and later) among others (configure.in lines 348–356). When it's on, SAFE_ACCEPT(x) runs x only if the listener list has a next entry (ap_listeners->next). The code that creates the mutex is also wrapped in SAFE_ACCEPT (prefork.c lines 1031–1035), so with one Listen there is no mutex at all. How macros expand in C is in Part 2.

Example 2: starting prefork with the defaults

=== 3. start with defaults, count processes and threads ===
--- prefork
  PID / PPID / threads(NLWP) / user
  27 1 1 root
  28 27 1 daemon
  30 27 1 daemon
  31 27 1 daemon
  32 27 1 daemon
  33 27 1 daemon
  httpd processes: 6, threads in total: 6
  scoreboard, first 80 slots: W____...........................................................................
  scoreboard, all 256 slots = '_' 4  'W' 1  '.' 251
  BusyWorkers: 1
  IdleWorkers: 4
  log: prefork.c(1087): AH00165: Accept mutex: none (default: pthread)

=== 4. prefork accept mutex: one Listen vs two ===
  two Listens: prefork.c(1087): AH00165: Accept mutex: pthread (default: pthread)

The PIDs change from run to run. ps -eo pid,ppid,nlwp,user shows one parent running as root (PPID 1) with five children running as daemon, each with a single thread (NLWP 1). Five is the default for StartServers (server/mpm/prefork/mpm_default.h line 33). The scoreboard lines come from mod_status's /server-status?auto, and we read them in section 6. The log: lines are what prefork_run() logs at LogLevel debug (prefork.c lines 1087–1092). With just Listen 8080 the mutex is none; adding Listen 8081 produced a pthread mutex. Exactly what the SAFE_ACCEPT code says.

3. worker — one listener thread and many worker threads

prefork needs as many processes as there are connections being handled at once. Processes each use their own memory, so hundreds of connections get heavy. worker solves this by putting many threads inside each child process. Each child has ThreadsPerChild (default 25, server/mpm/worker/mpm_default.h line 51) worker threads plus one listener thread that only accepts.

The two yellow boxes are data structures shared between threads. The socket travels from the listener to a worker thread in the order ① to ④.

The listener: secure an idle worker first, then accept

Stripped down to the main line, the listener thread's loop looks like this.

        if (!have_idle_worker) {
            rv = ap_queue_info_wait_for_idler(worker_queue_info, NULL);
            /* ... */
            have_idle_worker = 1;
        }
        /* ... accept mutex, poll if there are several Listens ... */
            rv = lr->accept_func(&csd, lr, ptrans);
        /* ... */
            if (csd != NULL) {
                rv = ap_queue_push_socket(worker_queue, csd, NULL, ptrans);

server/mpm/worker/worker.c lines 585–699

The order matters. The listener accepts only after making sure an idle worker thread exists (line 586). If every worker is busy it doesn't accept, it waits, so new clients stay in the operating system's queue (the backlog). A connection is never accepted with nobody to handle it — but the client waits that much longer. You can see this in the example in section 4.

Worker threads: pop from the queue and handle

        rv = ap_queue_pop_socket(worker_queue, &csd, &ptrans);
        /* ... */
        process_socket(thd, ptrans, csd, process_slot, thread_slot, bucket_alloc);

server/mpm/worker/worker.c lines 796–829

process_socket() calls ap_run_create_connection() and ap_process_connection() just like prefork (lines 482–499). The unit that takes a connection changed from process to thread, but the worker is tied up until the connection ends, same as prefork.

The bridge between them: server/mpm_fdqueue.c

The listener and the worker threads are connected by two data structures in server/mpm_fdqueue.c, shared by worker and event.

• fd_queue_t — a ring buffer of sockets (an array that wraps around to the start when it reaches the end) plus a mutex and a condition variable (a device for putting threads to sleep and waking them) (server/mpm_fdqueue.h lines 76–87). Its size is ThreadsPerChild (worker.c line 897).
• fd_queue_info_t — the number of currently idle worker threads (idlers) and a list of used pools kept for reuse (server/mpm_fdqueue.c lines 31–45).

The pushing side is short.

apr_status_t ap_queue_push_socket(fd_queue_t *queue,
                                  apr_socket_t *sd, void *sd_baton,
                                  apr_pool_t *p)
{
    fd_queue_elem_t *elem;
    apr_status_t rv;

    if ((rv = apr_thread_mutex_lock(queue->one_big_mutex)) != APR_SUCCESS) {
        return rv;
    }

    AP_DEBUG_ASSERT(!queue->terminated);
    AP_DEBUG_ASSERT(!ap_queue_full(queue));

    elem = &queue->data[queue->in++];
    if (queue->in >= queue->bounds)
        queue->in -= queue->bounds;
    elem->sd = sd;
    elem->sd_baton = sd_baton;
    elem->p = p;
    queue->nelts++;

    apr_thread_cond_signal(queue->not_empty);

    return apr_thread_mutex_unlock(queue->one_big_mutex);
}

server/mpm_fdqueue.c lines 382–407

It writes the socket and pool at position in and wraps to 0 when it reaches bounds (the size). Then it wakes one waiting worker thread through the not_empty condition variable. The popping side, ap_queue_pop_something(), sleeps on that condition variable when the queue is empty (lines 432–490). The comment on the function above states the precondition that "ap_queue_info_wait_for_idler has already been called to reserve an idle worker thread" (lines 376–381), and thanks to that, whether the queue is full (ap_queue_full) is only asserted in debug builds.

Easy to get wrong: 27 threads, not 25

Run the example and each worker child has 27 threads.

--- worker
  PID / PPID / threads(NLWP) / user
  64 1 1 root
  66 64 27 daemon
  67 64 27 daemon
  69 64 27 daemon
  httpd processes: 4, threads in total: 82

That's 25 worker threads + 1 listener + the main thread the process had from the start. The main thread sits in child_main() waiting for signals from the parent (the "pipe of death" in section 5) (worker.c lines 1298–1300). There is one more thread, start_threads, that creates the others, but it exits once it's done (worker.c line 1058). The three children are worker's default StartServers of 3 (mpm_default.h line 33). event has the same shape; the gdb check is in section 4.

4. event — taking keep-alive connections off the threads

The problem worker leaves

In worker, a keep-alive connection holds a worker thread for up to KeepAliveTimeout until the next request comes. That thread actually does nothing but stare at one socket. Browsers often keep the connection open after getting a page, so most threads end up "busy doing nothing". The comment at the top of event.c describes exactly this: traditionally Apache keeps an entire child process or thread waiting for data from the client, and this MPM solves it with a dedicated thread that handles the listening sockets and all sockets in keep-alive state (server/mpm/event/event.c lines 17–39).

A worker thread handles one request and leaves a state behind

event uses the same queue (fd_queue_t) and worker-thread layout as worker. The difference is that the process_socket() its worker threads call runs ap_run_process_connection() directly instead of ap_process_connection(), and after it returns, looks at the connection's state to decide what to do next. As we saw in section 1, event answers "I'm an async MPM", so the http module takes the async path: it handles one request, writes a state into c->cs->state and returns (Part 9).

The states are listed in include/httpd.h (lines 1328–1348).

Here's the part that hands a connection over to keep-alive.

    if (cs->pub.state == CONN_STATE_KEEPALIVE) {
        ap_update_child_status(cs->sbh, SERVER_BUSY_KEEPALIVE, NULL);
        /* ... comment ... */
        cs->queue_timestamp = apr_time_now();
        notify_suspend(cs);

        /* Add work to pollset. */
        cs->pub.sense = CONN_SENSE_DEFAULT;
        update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ);
        apr_thread_mutex_lock(timeout_mutex);
        TO_QUEUE_APPEND(cs->sc->ka_q, cs);
        rv = apr_pollset_add(event_pollset, &cs->pfd);
        /* ... failure handling ... */
        return;
    }

server/mpm/event/event.c lines 1305–1339

It appends the connection to the keep-alive queue (TO_QUEUE_APPEND), adds the socket to the pollset, and returns. Once the function returns, the worker thread goes back to the queue for its next job. The connection now belongs to no thread. Simply appending in time order is enough because each queue has one fixed waiting time — that's what the shortened comment above and the comment at the queue declarations (lines 273–286) explain.

The listener thread: polls listening sockets and waiting connections together

event's listener does far more than worker's. The listening sockets and the connection sockets handed over above all live in the same pollset (the listening sockets are registered as PT_ACCEPT at lines 2763–2772). The listener waits on both with a single apr_pollset_poll() (line 2112) and, when it wakes up, looks at what's ready. For a connection socket (PT_CSD) it takes the connection off the queue matching the state it was waiting in.

                switch (cs->pub.state) {
                case CONN_STATE_WRITE_COMPLETION:
                    remove_from_q = cs->sc->wc_q;
                    blocking = 1;
                    break;

                case CONN_STATE_ASYNC_WAITIO:
                    cs->pub.state = CONN_STATE_PROCESSING;
                    remove_from_q = cs->sc->io_q;
                    blocking = 1;
                    break;

                case CONN_STATE_KEEPALIVE:
                    cs->pub.state = CONN_STATE_PROCESSING;
                    remove_from_q = cs->sc->ka_q;
                    break;

                case CONN_STATE_LINGER_NORMAL:
                    remove_from_q = linger_q;
                    break;

                case CONN_STATE_LINGER_SHORT:
                    remove_from_q = short_linger_q;
                    break;

server/mpm/event/event.c lines 2149–2172

When the next request arrives on a keep-alive connection, the state becomes PROCESSING, the connection is taken off the queue and the pollset, and it's passed to an idle worker thread (push2worker(), lines 2208–2215). The thread that receives it doesn't have to be the one that first handled the connection. This is exactly what the header comment in ap_mpm.h foresaw: "in the future it will be possible for ap_process_connection to return to the MPM prior to finishing the entire connection … but does not guarantee it will occur on the same thread as the first call" (lines 49–53). For a listening socket (PT_ACCEPT) it accepts and passes the new connection on the same way (lines 2218–2297).

The listener also cleans up connections whose time is up. Once the earliest expiry at the head of the queues has passed, it closes expired connections on the keep-alive queue and processes the write-wait and closing queues in turn (lines 2352–2399).

The yellow box (the pollset) looks after "waiting connections". The green box, where the worker thread is freed, is the difference from worker.

Example 3: six keep-alive connections against four worker threads

To make the difference visible, I cut the worker threads down to four (StartServers 1, ServerLimit 1, ThreadsPerChild 4, MaxRequestWorkers 4). Then I used bash's /dev/tcp to open six connections, send one request on each, and leave them open — imitating a browser that got its response and hasn't sent another request. One second later curl sends one new request and we time it. (Below is a shortened excerpt of run.sh.)

# An example written for this post (not part of httpd's source)
for i in 3 4 5 6 7 8; do
  eval "exec $i<>/dev/tcp/127.0.0.1/8080"            # open a connection
  printf 'GET /index.html HTTP/1.1\r\nHost: localhost\r\n\r\n' >&$i   # one request, never closed
done
sleep 1
curl -s -o /dev/null -w "  time for a new request: %{time_total}s (HTTP %{http_code})\n" http://127.0.0.1:8080/index.html

=== 5. six idle keep-alive connections against 4 worker threads ===
--- worker (KeepAliveTimeout 5)
  time for a new request: 5.669387s (HTTP 200)
  BusyWorkers: 4
  IdleWorkers: 0
  Scoreboard: CKWK
--- event (KeepAliveTimeout 5)
  time for a new request: 0.001105s (HTTP 200)
  BusyWorkers: 1
  IdleWorkers: 3
  ConnsTotal: 0
  ConnsAsyncKeepAlive: 0
  Scoreboard: W___
  log: event.c(2015): connections: 6 (waitio:0 write-completion:0keep-alive:6 lingering:0 suspended:0 clogged:0), workers: 0/4 shutdown
--- worker (KeepAliveTimeout 2)
  time for a new request: 2.766733s (HTTP 200)
  BusyWorkers: 4
  IdleWorkers: 0
  Scoreboard: WKKC

Timings and the order of letters on the scoreboard vary a little from run to run.

• With worker the new request took 5.7 seconds. The first four connections held the four worker threads in keep-alive, so the listener didn't even accept until a thread became idle (section 3). Lowering KeepAliveTimeout to 2 seconds brought it down to 2.8 seconds, so the wait follows the keep-alive time. On the scoreboard K is a thread waiting for the next keep-alive request and C is a thread closing a connection. (I didn't look into why the wait is a bit under a second longer than KeepAliveTimeout — 0.6 to 0.9 seconds, depending on the run.)
• With event it took 0.001 seconds. The scoreboard shows three of the four threads idle (_) and one handling this status request (W). Yet the six connections weren't dropped. The last line is the log event's listener writes every second at LogLevel trace6 (lines 2011–2030): all six connections are on the keep-alive queue, with four worker threads. There can be more connections than threads.

Easy to get wrong

• ConnsAsyncKeepAlive: 0 does not mean "no keep-alive connections". At the same moment the listener's log said keep-alive:6. The connection counts on the scoreboard that mod_status reads (ps->keep_alive, ps->connections) are updated only when the listener maintains the timeout queues (lines 2393–2398). In this example the earliest expiry was 5 seconds away, so maintenance hadn't run yet. The numbers in the log are read on the spot from atomic variables (variables that are safe to change from several threads at once) held by the listener.
• The missing space in write-completion:0keep-alive:6 in the log isn't garbled output; that's the string in the source. In C two string literals written side by side are joined into one, and the first one has no trailing space (lines 2016–2017). event.c on the 2.4.x branch has the same string.
• event also drops keep-alive connections when workers run short. If the listener tried to hand off work and no worker thread was idle (workers_were_busy), it closes the whole keep-alive queue regardless of expiry times (process_keepalive_queue(0), lines 2364–2366, lines 2400–2406). The function's comment says this is "so that they may connect to another process" (lines 1946–1957). Also, once the number of open connections (not counting those being closed) reaches roughly ThreadsPerChild + idle threads × AsyncRequestWorkerFactor, the child temporarily switches off its listening sockets and stops accepting (connections_above_limit(), lines 552–570). It doesn't take connections without limit.
• event has no accept mutex. worker.c has the same SAFE_ACCEPT and mutex as prefork (worker.c line 603), but SAFE_ACCEPT and AP_ACCEPT_MUTEX_TYPE don't appear even once in event.c. The same goes for event.c on the 2.4.x branch.
• trunk vs 2.4.x: the state list above (including CONN_STATE_ASYNC_WAITIO) and the ka_q/wc_q/io_q queue layout also exist in include/httpd.h and event.c on the 2.4.x branch. I did not check how much of it is in each 2.4 release people usually install.

Example 4: what are an event child's 27 threads doing?

Inside the container there's no permission (ptrace) to attach gdb to a running process, so I started event in httpd -X (a debug mode that runs as one process without children) under gdb, stopped it after 3 seconds, and printed the call stack of every thread. My first attempt was to attach to a running child with gdb -p, which failed with "Could not attach to process", so I switched to this.

# An example written for this post (not part of httpd's source)
(sleep 3; kill -TRAP $(pgrep -x httpd)) &
timeout 30 gdb -batch -ex "handle SIGPIPE nostop noprint" -ex run -ex "thread apply all bt" -ex kill \
    --args /opt/httpd/bin/httpd -X -f /tmp/event.conf > /tmp/gdb.txt 2>&1 || true

=== 6. what are the 27 threads of one event child doing (gdb, -X single-process mode) ===
  27 threads; functions seen in their call stacks:
        1  in child_main
        1  in listener_thread
       25  in worker_thread

One main thread in child_main(), one listener_thread(), and 25 worker_thread()s. That matches the 27 threads ps showed for an event child with the default settings (the output in Example 5, section 6). start_threads(), which creates them, makes the worker threads, then the listener, and exits (lines 2855–2891).

5. The parent process: count, grow and shrink the children every second

In all three MPMs the parent process handles no requests. Its only job is looking after the children, and that loop is almost the same shape in all three. In prefork it's the tail end of prefork_run() (prefork.c lines 1104–1196).

1. ap_wait_or_timeout() — checks without waiting (APR_NOWAIT) whether a child has exited, and if not, sleeps for one second (server/mpm_common.c lines 268, 280).
2. If a child exited, it marks that child's scoreboard slot SERVER_DEAD and skips maintenance this round.
3. If the second simply passed, it calls perform_idle_server_maintenance() (line 1195).

worker and event do the same in server_main_loop() (for event, lines 3496–3607).

Every second the parent looks at the scoreboard and decides. It never asks the children directly.

perform_idle_server_maintenance()

In prefork's version it first walks the scoreboard slots and counts children whose state is SERVER_READY or lower (idle, or just starting) (prefork.c line 863). Then it decides.

    if (idle_count > ap_daemons_max_free) {
        static int bucket_kill_child_record = -1;
        /* kill off one child... we use the pod because that'll cause it to
         * shut down gracefully, in case it happened to pick up a request
         * while we were counting
         */
        bucket_kill_child_record = (bucket_kill_child_record + 1) % retained->mpm->num_buckets;
        ap_mpm_pod_signal(retained->buckets[bucket_kill_child_record].pod);
        retained->idle_spawn_rate = 1;
    }
    else if (idle_count < ap_daemons_min_free) {
        /* ... if there's no free slot, log "server reached MaxRequestWorkers setting" once ... */
        else {
            /* ... */
            for (i = 0; i < free_length; ++i) {
                make_child(ap_server_conf, free_slots[i]);
            }
            /* the next time around we want to spawn twice as many if this
             * wasn't good enough, but not if we've just done a graceful
             */
            if (retained->hold_off_on_exponential_spawning) {
                --retained->hold_off_on_exponential_spawning;
            }
            else if (retained->idle_spawn_rate < MAX_SPAWN_RATE) {
                retained->idle_spawn_rate *= 2;
                if (retained->idle_spawn_rate > MAX_SPAWN_RATE) {
                    retained->idle_spawn_rate = MAX_SPAWN_RATE;
                }
            }
        }
    }

server/mpm/prefork/prefork.c lines 872–919

• If more children are idle than MaxSpareServers, it removes only one per round, using ap_mpm_pod_signal(). The POD (pipe of death) is a pipe shared by parent and children; when the parent writes a byte into it, whichever child reads it first takes it as "my turn to quit" (server/mpm_unix.c lines 447–466, line 759). Why not kill a particular child with a signal? The comment says: that child might have picked up a request while we were counting, so let it finish and shut down gracefully. child_main() from section 2 checks this pipe after every connection it handles (prefork.c lines 702–704).
• If fewer than MinSpareServers, it creates children in free slots. The number per round (idle_spawn_rate) starts at 1 and doubles each time it's still not enough, up to MAX_SPAWN_RATE (32, line 131). When clients suddenly pour in, one per second is too slow, and creating many from the start is wasteful. Once it reaches 8 it logs "server seems busy" at info level (lines 895–902).
• If it's in between, it does nothing and resets the rate to 1.

worker and event use the same frame, but they count threads rather than children (MinSpareThreads/MaxSpareThreads), and removing one child takes ThreadsPerChild threads away at once (worker.c lines 1565–1569, event.c lines 3357–3394). event's version adds a condition that holds off shrinking if some children are already stopping and the scoreboard has no slack (lines 3374–3376).

Easy to get wrong: StartServers only matters at startup

StartServers is just the number passed to startup_children() at startup (prefork.c line 1070) (on a graceful restart the same number is used to replace exiting children one-for-one right away, lines 1065–1068 and 1151–1158). After that, Min/MaxSpare take over every second, so even if you start with StartServers 50, with no clients the children drop by one per second down to MaxSpareServers — because the code above only removes one per round.

6. The scoreboard — shared memory the parent and children both see

How does the parent know which children are idle? Processes can't see each other's memory (see "Words to know first"), so there's one status board in shared memory everyone can see, and each writes its own state into its own slot. That's the scoreboard.

The single yellow box is the whole shared memory. Writers only touch their own slot, so no lock is needed.

Three structs

include/scoreboard.h has three layers.

• global_score — only one. It holds the limits and the generation number that should currently be alive (running_generation) (lines 123–133). The generation goes up by one on every restart (Part 5), and a child whose generation differs from it quits (e.g. prefork.c lines 706–712).
• process_score — one per child process. PID, generation, whether it's on its way out (quiescing), and connection counts for async MPMs like event (lines 137–152). On the 2.4.x branch it also has a bucket field that is no longer used.
• worker_score — one per worker (a child in prefork, a thread in worker/event). State (status), number of requests handled, the start of the request line being handled (request[64]) and so on (lines 90–121).

status runs from SERVER_DEAD (0) to SERVER_IDLE_KILL (10) (lines 56–67). mod_status shows each as a single character (modules/generators/mod_status.c lines 1006–1017).

(6 L, 7 D and 10 I are left out of the table.) The maintenance code in section 5 counting "SERVER_READY or lower" as idle is meant to count STARTING (1) and READY (2) together, and the comment notes that this relies on the order of the constants (prefork.c lines 857–862).

Sized by ServerLimit × ThreadLimit, created by the parent

AP_DECLARE(int) ap_calc_scoreboard_size(void)
{
    ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, &thread_limit);
    ap_mpm_query(AP_MPMQ_HARD_LIMIT_DAEMONS, &server_limit);

    scoreboard_size  = SIZE_OF_global_score;
    scoreboard_size += SIZE_OF_process_score * server_limit;
    scoreboard_size += SIZE_OF_worker_score * server_limit * thread_limit;

    return scoreboard_size;
}

server/scoreboard.c lines 138–148

ap_mpm_query() shows up here too. The core's scoreboard code doesn't know which MPM it's dealing with; it just asks for the limits and sizes itself accordingly. prefork answers the thread limit with HARD_THREAD_LIMIT (1) (prefork.c lines 86–88), so there's one worker_score per child. This is also why ServerLimit and ThreadLimit can't be changed by a restart: the size of the shared memory is fixed the first time (each MPM's check_config puts the first value back and warns, e.g. event.c lines 4190–4198).

The creator is ap_create_scoreboard(), which the core hangs on the pre_mpm hook (server/core.c line 6178). The MPM runs that hook before forking children, on startup and on non-graceful restarts (prefork.c lines 998–1002). Without ScoreBoardFile it creates anonymous shared memory with apr_shm_create() (server/scoreboard.c line 232). Children born by fork inherit that memory and look at the same place.

Why it's written without locks

Several processes write to one block of memory, yet there's no lock. A source comment explains: the update functions change only one slot, and only one party writes a given slot at a time (the child that owns it, or the parent noting that the child died) (server/scoreboard.c lines 344–353). The parent and mod_status only read, so they may occasionally see slightly stale values, but for maintenance that looks again every second that's good enough.

Example 5: reading the scoreboard

From the same run as the example in section 2, here's mod_status's ?auto output.

--- prefork
  scoreboard, first 80 slots: W____...........................................................................
  scoreboard, all 256 slots = '_' 4  'W' 1  '.' 251
--- worker
  scoreboard, first 80 slots: _________________________W_________________________________________________.....
  scoreboard, all 400 slots = '_' 74  'W' 1  '.' 325
  BusyWorkers: 1
  IdleWorkers: 74
--- event
  PID / PPID / threads(NLWP) / user
  180 1 1 root
  182 180 27 daemon
  183 180 27 daemon
  185 180 27 daemon
  httpd processes: 4, threads in total: 82
  scoreboard, first 80 slots: __________________________________________________W________________________.....
  scoreboard, all 400 slots = '_' 74  'W' 1  '.' 325

• prefork has 256 slots: prefork's default ServerLimit of 256 (prefork.c line 76) × one slot per child. Of the five children, one is handling this status request (W) and four are idle (_); the other 251 slots are empty (.).
• worker and event have 400 slots: the default ServerLimit of 16 × ThreadsPerChild 25. The shared memory has 16 × ThreadLimit (default 64) slots, but mod_status leaves out slots beyond ThreadsPerChild (mod_status.c lines 334–338, lines 656–668). Three children × 25 = 75 slots are alive, and one of them is W.

7. Where the directives go in the code

The numeric MPM directives are all registered with the AP_INIT_TAKE1 we saw in Part 7, and the handlers mostly just read a number with atoi() (only AsyncRequestWorkerFactor uses strtod(), since it accepts a fraction) into a static variable in the file (e.g. event's table, event.c lines 4509–4532). Making the values consistent with each other happens all at once in the check_config hook.

Easy to get wrong: same MaxRequestWorkers, different meaning

prefork's directive description is "Maximum number of children alive at the same time" (prefork.c lines 1593–1594), and event's is "Maximum number of threads alive at the same time" (event.c lines 4521–4522). Same name, but in prefork it's a number of processes and in worker/event a number of threads. That's why, in worker/event, if MaxRequestWorkers isn't a multiple of ThreadsPerChild, check_config rounds it down to a multiple and warns (worker.c lines 2313–2330). And in event this value is not "the number of connections that can be handled at once". As the example in section 4 showed, keep-alive connections don't occupy a thread, so there can be more open connections than this.

Same name, but prefork counts processes and worker·event count threads. The calculation was checked in worker.c lines 2311–2330 and event.c lines 4288–4306.

8. Listening sockets are opened by the parent and survive restarts

ap_set_listener(), which handles the Listen directive, only adds an ap_listen_rec to the ap_listeners list (server/listen.c line 1074); the socket is actually opened by ap_setup_listeners(), called from the MPM's open_logs hook (section 1). That hook runs in the parent before forking, so the children inherit the open sockets and all accept on the same ones. Ports below 1024, like 80, can only be opened by root — this is why children can still accept on port 80 after dropping to the daemon user (Part 5).

On restart, sockets aren't closed and reopened either.

1. ap_listen_pre_config(), called from each MPM's pre_config, moves the current list to old_listeners (server/listen.c lines 986–988).
2. While the config is read again, each Listen with the same address and port as an old entry moves that entry (open socket and all) to the new list as it is (find_listeners() lines 424–456, lines 472–478).
3. When the sockets are opened, only the old sockets that aren't in the new config are closed (lines 699–701).

The three steps above attached to the startup flow. The yellow boxes happen only on restart.

So even during a restart, a listening socket on the same address is never closed, and clients arriving in the meantime wait in the operating system's queue for the new children.

Easy to get wrong: listener "buckets"

Names like my_bucket and num_buckets show up all over the source. If you set ListenCoresBucketsRatio and the OS supports SO_REUSEPORT, the listening sockets are duplicated in proportion to the number of CPU cores and the children are spread over those groups (buckets) (server/listen.c lines 823–870). The default is 0 (line 994), so there's one bucket, and all the examples in this post ran with one bucket. When you see something like % num_buckets in the code, read it as "divide by 1 with the default settings".

What this post leaves out

• Reaping children and threads on graceful restart and shutdown — ap_reclaim_child_processes() (server/mpm_unix.c), worker/event's signal_threads() and join_workers(), GracefulShutdownTimeout.
• event's timers and callbacks — ap_mpm_register_timed_callback(), ap_mpm_register_poll_callback() (include/ap_mpm.h lines 204–254, event.c lines 1581–1776). Modules like mod_proxy_wstunnel use these to hand connections over to the MPM.
• Lingering close, closing a connection slowly — ap_lingering_close() (server/connection.c), event's process_lingering_close() (event.c line 1778).
• CONN_STATE_SUSPENDED and the suspend/resume hooks (event.c lines 874–893, line 1354).
• Deciding to stop the server when children keep dying — sick_child_detected, APEXIT_CHILDSICK/APEXIT_CHILDFATAL.
• mod_unixd's signal setup, ap_unixd_mpm_set_signals() (os/unix/unixd.c).
• The Windows winnt MPM (server/mpm/winnt/) and the trunk-only motorz and simple.

Summary

What's next

We've followed the MPM up to handing a socket to a worker. What happens after the worker calls ap_process_connection() with that socket — the connection becoming a conn_rec, then the request line and headers being read into a request_rec — continues in Part 9: Connections and Reading a Request — From a Socket to a request_rec. The insides of the two sync/async functions that event only looked at by their result state in section 4 are in that post too.

Compared with the Spring series: worker's "one connection = one thread" resembles a servlet container thread taking one request and following it all the way through DispatcherServlet (Spring Part 21), and event taking waiting connections off the threads so a few threads watch them with poll targets the same problem as WebFlux's design of "not holding a thread while waiting for the whole body to arrive" (Spring Part 24). The difference is that event still uses one worker thread while a request is being processed; it only takes the waiting time between requests off the thread.

• Part 1: How I Read It with AI, and the Map
• Part 3: The Big Picture — Core, APR, MPM, Modules and the Three Big Structs
• Part 5: main() and Startup — Why the Config Is Read Twice, and Restarts
• Part 6: Modules and Hooks — How a Module Plugs into the Server
• How a Java server answers the same question: Digging into Tomcat's Source 5: The Connector and the NIO Endpoint — Which Threads Take a Connection
• A C server where one worker thread both watches connections and processes them, and new connections are shared out by default with EPOLLEXCLUSIVE instead of an accept mutex: Digging into nginx's Source 7: The Event Loop — epoll, Timers, Posted Events and accept