Digging into Apache httpd's Source 5: main() and Startup — Why the Config Is Read Twice, and Restarts

What this post covers

This is the fifth post in the "Digging into Apache httpd's Source" series. Part 4 showed how httpd manages memory with pools instead of free(). That post mentioned that names like pconf, ptemp and plog are created in server/main.c and moved on. This time we read that file from top to bottom. It is where the main() function of the httpd program lives.

There are four questions to answer.

• When you type httpd -k start, in what order does main() do its work?
• Is it true that the config file (httpd.conf) is read twice, and if so, why?
• When you run apachectl graceful or restart, what happens inside the server that is already running?
• Where do options like -t and -S stop, and where does a server started as root drop down to User?

Every line number and every detail in this post refers to that commit. Part 1 explains how to get the source at the same commit and the Docker setup used for the examples. Where the 2.4.x releases most people install differ, I say so as we go. To give it away early: the overall flow of main.c is almost the same on the 2.4.x branch.

main.c also contains code for special environments such as Windows, NetWare and EBCDIC, but this post follows only the main thread: Linux with the event MPM. What was left out is named at the end, under "What this post leaves out".

Words to know first

main() is the function where a C program starts. When you run httpd from a shell, the operating system calls it. httpd's main() is in server/main.c, a file of 914 lines.

Process and PID: a running program is a process, and the number the operating system gives it is its PID. A process can copy itself with fork() to create a child process. httpd is built as one parent process with root privileges looking after several child processes that handle requests. The part that decides this arrangement is the MPM (Multi-Processing Module), covered in Part 8.

A daemon is a process that runs in the background, detached from the terminal. When you type httpd -k start the prompt comes straight back, because in the meantime httpd turns itself into a daemon. Section 4 finds that exact moment in the source.

A pool is APR's way of grouping memory so it can be freed all at once. Two things matter in this post: clearing a pool (apr_pool_clear) releases everything allocated from it and runs the cleanup functions (cleanups) registered on it, all in one go; and the configuration lives in a pool called pconf. Pools themselves were covered in Part 4.

A hook is a spot where a module says "call this function of mine when we reach this point". This post meets the startup hooks pre_config, open_logs, post_config, child_init and a few others. These four call every registered function in turn (the RUN_ALL or VOID style), and a RUN_ALL hook stops as soon as one returns an error (server/config.c lines 88–104, lines 160–167). The macros that build hooks and the way their order is decided are in Part 6.

A signal is a short notification one Unix process sends to another. The kill command sends signals, and httpd understands SIGTERM (stop), SIGHUP (restart) and SIGUSR1 (restart gracefully).

1. main() at a glance

Reading main() (line 527) from top to bottom, it falls into five chunks.

Blue boxes are the steps that read the config and call hooks; green boxes are the early exits.

1. Prepare the process — init_process() (lines 342–410) initialises APR and creates the process pool, which lives until the process ends, and pconf underneath it. The process pool is also stored in the global variable ap_pglobal (line 551).
2. Register the built-in modules — ap_setup_prelinked_modules() (lines 574–579) registers the modules that are compiled into the httpd binary.
3. Parse the command line — options such as -f, -D and -t (lines 586–702).
4. First read — read the config file and call the hooks (lines 704–808). -t and -k stop end in here.
5. The loop — inside do { ... } while, read the config again and hand over to the MPM (lines 810–886). A restart is just one more trip round this loop.

Only three modules are built in

The list registered in step 2, ap_prelinked_modules, is not in the source tree. It lives in a file called modules.c that configure generates at build time. Here is that file from this series' example container (built with --enable-mods-shared=most).

AP_DECLARE_DATA module *ap_prelinked_modules[] = {
  &core_module,
  &so_module,
  &http_module,
  NULL
};

Part of /usr/src/httpd/modules.c, generated at build time. httpd -l shows the same three (core.c, mod_so.c, http_core.c).

The core, mod_so (which owns the LoadModule directive that loads .so files), and http_module for the HTTP protocol. Not even an MPM is in there. This build made the MPMs .so files too (--enable-mpms-shared=all), so without a LoadModule mpm_event_module ... line in httpd.conf the server cannot start. If no MPM has been loaded once the whole config is read, ap_read_config() reports an error (server/config.c lines 2355–2360). The big picture of how the core and the modules split up is in Part 3.

ap_setup_prelinked_modules() (server/config.c lines 763–816) gives each module a number (module_index) and registers it with ap_add_module(), which also calls the module's hook-registration function. Line 769 makes pconf the memory that holds the hooks (apr_hook_global_pool=process->pconf); this becomes important in section 3.

main() never sees -k

The option string passed to the apr_getopt() loop that parses the command line is "C:c:D:d:E:e:f:vVlLtTSMh?X" (include/http_main.h line 36). There is no k in it. httpd -k start still works thanks to ap_run_rewrite_args(process), called just before (line 581).

The core module's struct has a slot for "a function to call before the command line is parsed", and on Unix that slot holds ap_mpm_rewrite_args() (server/core.c line 6218, os/unix/os.h line 45). This function picks out only -k start|stop|restart|graceful|graceful-stop, keeps the value aside, builds a new command line from the remaining arguments and swaps it into process->argv (server/mpm_unix.c lines 900–955). So main()'s apr_getopt() never sees -k. The saved -k value comes back at the end of section 2.

Most options do very little. Here are -t and -S.

        case 't':
            if (ap_run_mode == AP_SQ_RM_UNKNOWN)
                ap_run_mode = AP_SQ_RM_CONFIG_TEST;
            break;

       case 'T':
           ap_document_root_check = 0;
           break;

        case 'S':
            ap_run_mode = AP_SQ_RM_CONFIG_DUMP;
            new = (const char **)apr_array_push(ap_server_config_defines);
            *new = "DUMP_VHOSTS";
            new = (const char **)apr_array_push(ap_server_config_defines);
            *new = "DUMP_RUN_CFG";
            break;

server/main.c lines 656–671

-t only switches the run mode (ap_run_mode) to "config test", and -S is handled exactly as if you had added -D DUMP_VHOSTS -D DUMP_RUN_CFG. -M likewise becomes -D DUMP_MODULES (lines 673–677). The other way round, if you type -D DUMP_RUN_CFG yourself, lines 607–622 switch the run mode to "config dump". Where these values are used is in section 6.

2. The first read — the order in which the config is read and hooks are called

Once the command line has been parsed, main() sets its state to AP_SQ_MS_CREATE_PRE_CONFIG and creates two more pools.

    ap_main_state = AP_SQ_MS_CREATE_PRE_CONFIG;
    apr_pool_create(&plog, ap_pglobal);
    apr_pool_tag(plog, "plog");
    apr_pool_create(&ptemp, pconf);
    apr_pool_tag(ptemp, "ptemp");

    /* Note that we preflight the config file once
     * before reading it _again_ in the main loop.
     * This allows things, log files configuration
     * for example, to settle down.
     */

server/main.c lines 704–714

• plog holds things that are reopened every time the config is read, such as log files. Its parent is not pconf but the process pool (ap_pglobal).
• ptemp is a scratch pool used only while the config is being read; it is a child of pconf. When reading is done it is thrown away whole at line 808.

And the comment is the first answer to this post's question: "we preflight the config file once before reading it again in the main loop. This allows things, log files configuration for example, to settle down." Section 4 checks in the source what that actually means.

The first read goes in this order.

1. ap_read_config() (line 721) — reads the config file into a directive tree. Only directives that must run the moment they are read (EXEC_ON_READ), such as LoadModule, are executed at this point. That way, directives belonging to a module that was just loaded can be recognised further down (modules/core/mod_so.c line 449).
2. The pre_config hook (lines 742–746) — modules are told "the config is about to be processed, get your defaults ready". For example, mod_unixd resets User and Group to their defaults here (section 7).
3. ap_process_config_tree() (lines 748–749) — walks the tree and runs all the remaining directives. That process is the subject of Part 7.
4. The check_config hook (lines 760–764) — modules check that the config values make sense together.
5. With -t or -S it ends here, and so does sending a signal such as -k stop (code below).
6. The open_logs hook (lines 794–800) — plog is cleared just before, then the log files are opened. The core's ap_open_logs() redirects standard error (stderr) to the ErrorLog file here (server/log.c lines 405–438).
7. The post_config hook (lines 802–806) — the place where modules finish up once the config is complete, for example by creating shared memory.

The seven steps of the list above and the exits at step 5, in one line. Fall into the red box and httpd ends there.

The two exits in step 5 look like this.

        if (ap_run_mode != AP_SQ_RM_NORMAL) {
            if (showdirectives) { /* deferred in case of DSOs */
                ap_show_directives();
                destroy_and_exit_process(process, 0);
            }
            else {
                ap_run_test_config(pconf, ap_server_conf);
                if (ap_run_mode == AP_SQ_RM_CONFIG_TEST)
                    ap_log_error(APLOG_MARK, APLOG_STARTUP, 0, NULL, "Syntax OK");
            }
            destroy_and_exit_process(process, 0);
        }
    }

    /* If our config failed, deal with that here. */
    if (rv != OK) {
        destroy_and_exit_process(process, 1);
    }

    signal_server = APR_RETRIEVE_OPTIONAL_FN(ap_signal_server);
    if (signal_server) {
        int exit_status;

        if (signal_server(&exit_status, pconf) != 0) {
            destroy_and_exit_process(process, exit_status);
        }
    }

server/main.c lines 766–792

If the run mode is not "normal", as with -t or -S, it calls the test_config hook (that is where dumps get printed) and exits. In normal mode it looks up a function named ap_signal_server and calls it. That function was registered by section 1's ap_mpm_rewrite_args() as an optional function (a function looked up by name, used if present and skipped if not) (server/mpm_unix.c line 954). If it returns non-zero, httpd ends right there. Section 5 looks at it in detail.

Easy to get wrong: typos are found after pre_config

"Reading the config" sounds like one event, but steps 1 and 3 above are separate. Step 1, ap_read_config(), only turns the file into a tree; whether directive names are valid is checked in step 3 as ap_process_config_tree() walks the tree. When it meets a name nobody registered, it returns "Invalid command" (server/config.c lines 1312–1327; the error is printed at lines 1961–1989). So even a config with a typo gets the pre_config hook called once. The last step of the example in section 3 shows this for real.

3. do { ... } while — the second read, and what a restart really is

After the first read, main() enters the loop. The part that calls the hooks is long, so it is trimmed.

    do {
        ap_main_state = AP_SQ_MS_DESTROY_CONFIG;
        reset_process_pconf(process);

        ap_main_state = AP_SQ_MS_CREATE_CONFIG;
        ap_config_generation++;
        for (mod = ap_prelinked_modules; *mod != NULL; mod++) {
            ap_register_hooks(*mod, pconf);
        }

        /* This is a hack until we finish the code so that it only reads
         * the config file once and just operates on the tree already in
         * memory.  rbb
         */
        ap_conftree = NULL;
        apr_pool_create(&ptemp, pconf);
        apr_pool_tag(ptemp, "ptemp");
        ap_server_root = def_server_root;
        ap_server_conf = NULL; /* set early by ap_read_config() for logging */
        /* ... ap_read_config(), pre_config, ap_process_config_tree(),
         *     check_config, open_logs, post_config — same order as the first read ... */

        apr_pool_destroy(ptemp);
        apr_pool_lock(pconf, 1);

        ap_run_optional_fn_retrieve();

        ap_main_state = AP_SQ_MS_RUN_MPM;
        rc = ap_run_mpm(pconf, plog, ap_server_conf);

        apr_pool_lock(pconf, 0);

    } while (rc == OK);

    if (rc == DONE) {
        rc = OK;
    }
    else if (rc != OK) {
        ap_log_error(APLOG_MARK, APLOG_EMERG, 0, NULL, APLOGNO(02818)
                     "MPM run failed, exiting");
    }
    destroy_and_exit_process(process, rc);

server/main.c lines 810–895

The body of the loop is almost the same as the first read. The differences are at the start and the end.

• Start: reset_process_pconf() clears pconf, the config generation number (ap_config_generation) goes up by one, and the hooks of the three built-in modules are registered again.
• End: ap_run_mpm() hands control to the MPM. The MPM starts the child processes and does not return from this function until a signal arrives.

In the yellow box the previous generation's configuration disappears entirely. A restart is following the arrow back to that box.

What ap_run_mpm() returns decides the loop's fate. The comment on the hook's declaration is short and clear: "It will run until a restart/shutdown is indicated. @return DONE for shutdown OK otherwise." (include/ap_mpm.h lines 83–93) If it is OK (0), while (rc == OK) is true and the loop goes round again. That is a restart. If it is DONE (-2), the loop ends and the process exits normally.

In other words httpd has no separate "restart" code. The first trip round the loop at startup and another trip round it on a restart are the same code.

What disappears when pconf is cleared

Here is reset_process_pconf(), called at the top of the loop.

/* Deregister all hooks when clearing pconf (pre_cleanup).
 * TODO: have a hook to deregister and run them from here?
 *       ap_clear_auth_internal() is already a candidate.
 */
static apr_status_t deregister_all_hooks(void *unused)
{
    (void)unused;
    ap_clear_auth_internal();
    apr_hook_deregister_all();
    return APR_SUCCESS;
}

static void reset_process_pconf(process_rec *process)
{
    if (process->pconf) {
        apr_pool_clear(process->pconf);
        ap_server_conf = NULL;
    }
    else {
        apr_pool_create(&process->pconf, process->pool);
        apr_pool_tag(process->pconf, "pconf");
    }
    apr_pool_pre_cleanup_register(process->pconf, NULL, deregister_all_hooks);
}

server/main.c lines 317–340

If pconf already exists it is cleared (apr_pool_clear), and deregister_all_hooks is registered as a pre-cleanup (a cleanup that runs first when the pool is cleared). So clearing pconf brings all of this with it:

• Every config structure allocated from pconf is gone — the previous generation's server_rec and every module's configuration.
• Every registered hook is removed (apr_hook_deregister_all()). That is why the loop registers the built-in modules' hooks again at lines 816–818. The hooks of .so modules are registered again when LoadModule runs again.
• The .so files loaded with LoadModule are unloaded. When mod_so loads a module it registers a cleanup on the config pool (modules/core/mod_so.c lines 309–314), and APR registers a cleanup that calls dlclose() on the pool the .so handle was opened in (APR dso/unix/dso.c lines 58–78, line 162).

So after a restart the server is one that has read its config file from scratch. That is also why removing a LoadModule line from httpd.conf and doing a graceful restart drops that module.

To remember something across restarts, attach it to the process pool, like the green box.

What about values that must survive a restart? httpd provides ap_retained_data_create() and ap_retained_data_get(). They attach memory to the process pool (ap_pglobal) under a key name (server/config.c lines 2526–2541). This is how the MPMs remember "which generation is this" and "was it graceful".

Example: how many times, and in which process, are the startup hooks called?

Let's see it. I wrote a module that writes one log line from each of the four startup hooks and from a cleanup on pconf. This example was run with httpd 2.5.1-dev built unchanged from the reference commit (e4af30fe6d) in a Docker container (Ubuntu 24.04). The code below is the example file mod_startlog.c with the repetitive hook functions and the #include lines trimmed.

/* Example written for this post (not httpd source) */
/* ... the file's header comment and #includes ... */

static int calls_static = 0;     /* an ordinary static variable inside the .so */

typedef struct {
    int calls;                   /* a counter that survives restarts */
} startlog_retained;

/* ... state_name(): ap_state_query(AP_SQ_MAIN_STATE) as a name ... */

static void note(const char *hook)
{
    startlog_retained *r = ap_retained_data_get("startlog");
    FILE *f;

    if (r == NULL) {
        /* created only once — it hangs off the process pool (ap_pglobal) */
        r = ap_retained_data_create("startlog", sizeof(*r));
    }
    r->calls++;
    calls_static++;

    f = fopen("/tmp/startlog.txt", "a");
    if (f) {
        fprintf(f, "%-11s pid=%-5d ppid=%-5d uid=%d state=%-17s gen=%d"
                   " static=%d retained=%d\n",
                hook, (int)getpid(), (int)getppid(), (int)getuid(),
                state_name(), ap_state_query(AP_SQ_CONFIG_GEN),
                calls_static, r->calls);
        fclose(f);
    }
}

/* cleanup called when pconf is cleared */
static apr_status_t pconf_cleared(void *data)
{
    note("(pconf cleanup)");
    return APR_SUCCESS;
}

static int startlog_pre_config(apr_pool_t *pconf, apr_pool_t *plog,
                               apr_pool_t *ptemp)
{
    note("pre_config");
    apr_pool_cleanup_register(pconf, NULL, pconf_cleared,
                              apr_pool_cleanup_null);
    return OK;
}

/* ... open_logs, post_config and child_init are each one note("hook name") line;
 *     register_hooks() hooks all four with APR_HOOK_MIDDLE ... */

The config uses the event MPM, User daemon and two child processes (StartServers 2). run.sh runs httpd -k start → -k graceful → -k restart → -k stop in turn and runs ps in between. Inside the container the shell running run.sh is PID 1, so ppid=1 means "the shell is the parent" or "the parent has already exited". PIDs may differ from run to run, and lines written by several processes at once can come out in a slightly different order each time (on other runs of the same example, a child_init line and a (pconf cleanup) line swapped places, or the two child_init lines did).

== httpd -k start  (shell pid 1)
pre_config  pid=153   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=1 retained=1
open_logs   pid=153   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=2 retained=2
post_config pid=153   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=3 retained=3
(pconf cleanup) pid=153   ppid=1     uid=0 state=other             gen=0 static=4 retained=4
pre_config  pid=154   ppid=153   uid=0 state=CREATE_CONFIG     gen=1 static=1 retained=5
open_logs   pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=1 static=2 retained=6
post_config pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=1 static=3 retained=7
child_init  pid=155   ppid=154   uid=1 state=RUN_MPM           gen=1 static=4 retained=8
child_init  pid=156   ppid=154   uid=1 state=RUN_MPM           gen=1 static=4 retained=8

A single httpd -k start called pre_config, open_logs and post_config twice each. The first time the state is CREATE_PRE_CONFIG (the first read), the second time CREATE_CONFIG (the second read inside the loop). In between, pconf was cleared once and the generation number (gen) went from 0 to 1. Two more things are worth noticing.

• The PID changed from 153 to 154. The first read was done by 153, which the shell started; from the second read on it is 154, whose parent is 153. That is section 4's topic.
• static went back to 1 while retained kept counting. Explained under "Easy to get wrong" below.

Next come graceful, restart and stop.

== httpd -k graceful
pre_config  pid=214   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=1 retained=1
(pconf cleanup) pid=154   ppid=1     uid=0 state=other             gen=1 static=4 retained=8
pre_config  pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=2 static=1 retained=9
open_logs   pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=2 static=2 retained=10
post_config pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=2 static=3 retained=11
(pconf cleanup) pid=214   ppid=1     uid=0 state=other             gen=0 static=2 retained=2
child_init  pid=216   ppid=154   uid=1 state=RUN_MPM           gen=2 static=4 retained=12
child_init  pid=217   ppid=154   uid=1 state=RUN_MPM           gen=2 static=4 retained=12
== httpd -k restart
pre_config  pid=273   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=1 retained=1
(pconf cleanup) pid=154   ppid=1     uid=0 state=other             gen=2 static=4 retained=12
pre_config  pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=3 static=1 retained=13
open_logs   pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=3 static=2 retained=14
post_config pid=154   ppid=1     uid=0 state=CREATE_CONFIG     gen=3 static=3 retained=15
(pconf cleanup) pid=273   ppid=1     uid=0 state=other             gen=0 static=2 retained=2
child_init  pid=275   ppid=154   uid=1 state=RUN_MPM           gen=3 static=4 retained=16
child_init  pid=276   ppid=154   uid=1 state=RUN_MPM           gen=3 static=4 retained=16
== httpd -k stop
pre_config  pid=332   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=1 retained=1
(pconf cleanup) pid=332   ppid=1     uid=0 state=other             gen=0 static=2 retained=2
(pconf cleanup) pid=154   ppid=1     uid=0 state=other             gen=3 static=4 retained=16

Two kinds of process are mixed together in this log.

• The freshly started httpd -k graceful process (214, 273, 332) calls pre_config once and exits. It never gets as far as open_logs or post_config. That is section 2's lines 785–792, the exit that just sends a signal.
• The original parent, 154, clears pconf (ending a generation), goes round the loop once more calling pre_config, open_logs and post_config once each, and starts new children. gen climbs 1→2→3 and the child PIDs change from 155/156 to 216/217 to 275/276. Parent 154 stays the same throughout.

Here are the counts per hook and the matching lines from the parent's error log.

### error_log (notice lines)
[mpm_event:notice] [pid 154:tid 154] AH00489: Apache/2.5.1-dev (Unix) configured -- resuming normal operations
[mpm_event:notice] [pid 154:tid 154] AH00493: SIGUSR1 received.  Doing graceful restart
[mpm_event:notice] [pid 154:tid 154] AH00489: Apache/2.5.1-dev (Unix) configured -- resuming normal operations
[mpm_event:notice] [pid 154:tid 154] AH00494: SIGHUP received.  Attempting to restart
[mpm_event:notice] [pid 154:tid 154] AH00489: Apache/2.5.1-dev (Unix) configured -- resuming normal operations
[mpm_event:notice] [pid 154:tid 154] AH00491: caught SIGTERM, shutting down
### counts per hook
pre_config       7
open_logs        4
post_config      4
child_init       6
(pconf cleanup)  7

Four post_config calls = two at startup + one for graceful + one for restart. pre_config is 7 because the -k graceful, -k restart and -k stop processes add one call each.

Easy to get wrong: static variables do not survive a restart

The static value in the log is an ordinary static int inside the .so. It goes back to 1 every time pconf is cleared. As we saw above, clearing pconf unloads the .so with dlclose() and the second read's LoadModule loads it again. In this example (Linux, Ubuntu 24.04) the static variable started from its initial value again when that happened. By contrast retained, made with ap_retained_data_create(), hangs off the process pool and keeps counting from 1 to 16.

The children's static=4 is the parent's count of 3 (up to post_config) inherited through fork(), plus one. A child starts as a copy of the parent's memory.

A module that needs to remember something across restarts should use retained data, not a static variable. httpd's MPMs do exactly that (see event_pre_config() in section 4).

Where I got stuck: at first child_init printed nothing

On the first run the log had zero child_init lines. The cause was section 7's topic itself. The log file /tmp/startlog.txt was created by root (the shell), and by the time child_init is called the child processes have already dropped to the daemon user, so they had no permission to write to it. Because I had written the code to skip silently when fopen() fails, there was no error to see either. Adding chmod 666 after creating the file in run.sh produced the output above.

4. Why read the config twice?

As the example showed, httpd really does read the config twice at startup. Looking for the reason in the source, it splits into what the source says outright and the effects you can read off the source.

What the source says outright

There are explanations in three places.

• Lines 710–714, seen in section 2: the config file is preflighted once before being read again in the main loop, which allows things, log files configuration for example, to settle down.
• The descriptions of the state values: AP_SQ_MS_CREATE_PRE_CONFIG is the "initial configuration run for setting up log config, etc.", and AP_SQ_MS_CREATE_CONFIG the "normal configuration run" (include/http_core.h lines 1083–1095).
• Lines 820–823 inside the loop: "This is a hack until we finish the code so that it only reads the config file once and just operates on the tree already in memory." The comment sits on the code that throws away the old tree with ap_conftree = NULL and reads the file again.

So the developers themselves did not consider reading twice to be the ideal design. That "finishing" has not happened since, and main.c on the 2.4.x branch has the same structure and the same comments.

The effects you can read off the source — the moment it becomes a daemon

What "settle down" means in practice shows up on the MPM side. Here is the event MPM's pre_config hook.

    retained = ap_retained_data_get(userdata_key);
    if (!retained) {
        retained = ap_retained_data_create(userdata_key, sizeof(*retained));
        retained->mpm = ap_unixd_mpm_get_retained_data();
        retained->mpm->baton = retained;
        if (retained->mpm->module_loads) {
            test_atomics = 1;
        }
    }
    else if (retained->mpm->baton != retained) {
        /* If the MPM changes on restart, be ungraceful */
        retained->mpm->baton = retained;
        retained->mpm->was_graceful = 0;
    }
    retained->mpm->mpm_state = AP_MPMQ_STARTING;
    ++retained->mpm->module_loads;
    /* ... */

    /* sigh, want this only the second time around */
    if (retained->mpm->module_loads == 2) {
        rv = apr_pollset_create(&event_pollset, 1, plog,
                                APR_POLLSET_THREADSAFE | APR_POLLSET_NOCOPY);
        if (rv != APR_SUCCESS) {
            ap_log_error(APLOG_MARK, APLOG_CRIT, rv, NULL, APLOGNO(00495)
                         "Couldn't create a Thread Safe Pollset. "
                         "Is it supported on your platform?"
                         "Also check system or user limits!");
            return HTTP_INTERNAL_SERVER_ERROR;
        }
        apr_pollset_destroy(event_pollset);

        if (!one_process && !foreground) {
            /* before we detach, setup crash handlers to log to errorlog */
            ap_fatal_signal_setup(ap_server_conf, pconf);
            rv = apr_proc_detach(no_detach ? APR_PROC_DETACH_FOREGROUND
                                 : APR_PROC_DETACH_DAEMONIZE);
            if (rv != APR_SUCCESS) {
                ap_log_error(APLOG_MARK, APLOG_CRIT, rv, NULL, APLOGNO(00496)
                             "apr_proc_detach failed");
                return HTTP_INTERNAL_SERVER_ERROR;
            }
        }
    }

server/mpm/event/event.c lines 3993–4047

module_loads counts how many times this pre_config has been called; it lives in retained data, so the count carries across restarts. Only on the second call (the comment: "sigh, want this only the second time around") does it become a daemon with apr_proc_detach(). This hook is registered as APR_HOOK_REALLY_FIRST (line 4378), so it runs in the very first group of the second read's pre_config hooks. A few other hooks are also registered as REALLY_FIRST (the core's core_pre_config, server/core.c line 6160, mod_log_config and others), so it is not literally the first, but it does run before the example module's pre_config, which uses APR_HOOK_MIDDLE. You can see that in the example log: the second-read pre_config line was already written by the new PID 154.

What apr_proc_detach() does is in APR.

    if (daemonize) {
        int x;

        if ((x = fork()) > 0) {
            exit(0);
        }
        else if (x == -1) {
            perror("fork");
            fprintf(stderr, "unable to fork new process\n");
            exit(1);  /* we can't do anything here, so just exit. */
        }
        /* RAISE_SIGSTOP(DETACH); */
    }

APR threadproc/unix/procsup.c lines 28–40

It creates a child with fork() and the original process ends with exit(0). That is the moment the example's PID changed from 153 to 154. Because 153 calls exit(0), httpd -k start returns to the shell as a "success" and the prompt comes back.

The red exit exists only before becoming a daemon. Once it is a daemon, the shell has no way of knowing about a failure.

Here the role of the first read becomes clear.

1. Config errors can only be reported to the shell before becoming a daemon. The moment it daemonises, the original process unconditionally ends with exit(0), so whatever fails afterwards, the exit code of httpd -k start is 0. The first read happens inside the process the shell started, so a failure there can return exit code 1 through destroy_and_exit_process(process, 1) (lines 284–308).
2. -k stop and graceful need to know the config. The running server's PID is written in PidFile, and that path is in the config file. That is why sending signals (lines 785–792) comes after the first read.
3. The log files get a trial opening. open_logs is called in the first read too and tries to open ErrorLog, so a bad log path also fails before daemonising (Unable to open logs, lines 796–800).

Point 3 matches the "log files configuration" in the comment at lines 710–714, but the source does not say that points 1 and 2 were the original reason for reading twice. What the source establishes is the fact that "the first read happens before daemonising, in the process the shell started", and its consequences.

The last step of the example checks point 1. I misspelled StartServers as StartServer and ran -k start.

### 6) a typo in the config: httpd -k start
AH00526: Syntax error on line 12 of /tmp/bad.conf:
Invalid command 'StartServer', perhaps misspelled or defined by a module not included in the server configuration
exit status: 1
### /tmp/startlog.txt after the failed start
pre_config  pid=346   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=1 retained=1
(pconf cleanup) pid=346   ppid=1     uid=0 state=other             gen=0 static=2 retained=2
    PID COMMAND
(no httpd processes)

The error message appears on the terminal and the exit code is 1. And, as section 2 said, pre_config was called once despite the typo.

Easy to get wrong: post_config is called twice at startup

For anyone writing a module this is the most practical conclusion. If you create shared memory or start an external process in post_config, you will do it twice at startup. So httpd's modules include code to skip the first read. The example module mod_example_ipc spells out why in a comment.

    /*
     * Do nothing if we are not creating the final configuration.
     * The parent process gets initialized a couple of times as the
     * server starts up, and we don't want to create any more mutexes
     * and shared memory segments than we're actually going to use.
     */
    if (ap_state_query(AP_SQ_MAIN_STATE) == AP_SQ_MS_CREATE_PRE_CONFIG)
        return OK;

modules/examples/mod_example_ipc.c lines 127–134

"The parent process gets initialized a couple of times as the server starts up, and we don't want to create any more mutexes and shared memory segments than we're actually going to use." The event MPM's post_config (lines 4081–4084), mod_systemd, mod_proxy_balancer, mod_ssl's session cache and others do the same: in modules/ and server/ at the reference commit, 21 files besides main.c, in 23 places, check for AP_SQ_MS_CREATE_PRE_CONFIG.

5. Restarting and stopping — how one -k turns the loop

Now let's follow what passes between the process that ran -k graceful and the parent that was already running.

The sending side: ap_signal_server()

The -k value saved in section 1 is used by ap_signal_server() (server/mpm_unix.c lines 812–898). It reads the PID from PidFile, checks with kill(pid, 0) whether that process is alive, and sends a signal depending on the value. Here is the graceful part.

    if (!strcmp(dash_k_arg, "graceful")) {
        if (!running) {
            printf("httpd not running, trying to start\n");
        }
        else {
            *exit_status = send_signal(otherpid, AP_SIG_GRACEFUL);
            return 1;
        }
    }

server/mpm_unix.c lines 877–885

That AP_SIG_GRACEFUL is SIGUSR1 was checked in include/mpm_common.h line 76. The signal for graceful-stop comes from the definition in the same file; I did not run it in this post's example. If the function returns 1, main() ends there; if it returns 0 (for start, or when there is no server and it starts one instead), startup continues.

The receiving side: signal → the MPM's return value → main()'s loop

Grey is the short-lived -k process; blue is the parent that keeps running.

The parent's signal handlers only set flags. SIGHUP and SIGUSR1 go to sig_restart(), which sets restart_pending = 1, and, if the signal is not SIGUSR1 (that is, it is SIGHUP), also sets is_ungraceful = 1 (os/unix/unixd.c lines 486–504). SIGTERM makes sig_term() set shutdown_pending (lines 466–484).

The MPM's parent looks after its children in server_main_loop() and leaves it as soon as one of those flags is set (server/mpm/event/event.c line 3496). At the end of event_run() it decides what to return. For a shutdown it is DONE (line 3806, line 3868); for a restart it is this part.

    /* we've been told to restart */
    if (one_process) {
        /* not worth thinking about */
        return DONE;
    }

    if (!retained->mpm->is_ungraceful) {
        ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, ap_server_conf, APLOGNO(00493)
                     "%s received.  Doing graceful restart",
                     AP_SIG_GRACEFUL_STRING);
    }
    else {
        ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, ap_server_conf, APLOGNO(00494)
                     "SIGHUP received.  Attempting to restart");
    }
    return OK;

server/mpm/event/event.c lines 3871–3886

When OK reaches main(), the loop from section 3 goes round once more. The AH00493 and AH00494 lines in the example's error log were printed by this code. In -X mode (debug, a single process) it returns DONE even on a restart signal and simply exits (lines 3872–3875).

Where graceful and restart part ways

In the example both graceful and restart changed the child PIDs, so from the outside they looked the same. The difference is in how the new generation's event_run(), as it starts, sends off the previous generation's children.

    if (!retained->gen_pool) {
        apr_pool_create(&retained->gen_pool, ap_pglobal);
    }
    else {
        if (retained->mpm->was_graceful) {
            /* wake up the children...time to die.  But we'll have more soon */
            for (i = 0; i < num_buckets; i++) {
                ap_mpm_podx_killpg(retained->buckets[i].pod,
                                   active_daemons_limit, AP_MPM_PODX_GRACEFUL);
            }
        }
        else {
            /* Kill 'em all.  Since the child acts the same on the parents SIGTERM
             * and a SIGHUP, we may as well use the same signal, because some user
             * pthreads are stealing signals from us left and right.
             */
            for (i = 0; i < num_buckets; i++) {
                ap_mpm_podx_killpg(retained->buckets[i].pod,
                                   active_daemons_limit, AP_MPM_PODX_RESTART);
            }
            ap_reclaim_child_processes(1,  /* Start with SIGTERM */
                                       event_note_child_stopped);
        }
        apr_pool_clear(retained->gen_pool);
        retained->buckets = NULL;

        /* advance to the next generation */
        /* XXX: we really need to make sure this new generation number isn't in
         * use by any of the previous children.
         */
        ++retained->mpm->my_generation;
    }

server/mpm/event/event.c lines 3627–3658

• graceful (was_graceful is true): the children are told AP_MPM_PODX_GRACEFUL and not waited for. Each child finishes the request it is handling and leaves on its own.
• restart: after telling them AP_MPM_PODX_RESTART, it reaps the children with ap_reclaim_child_processes(1, ...), starting with SIGTERM. Requests in progress are cut off (in the ungraceful mode the child calls close_worker_sockets(), commented "forcefully kill all current connections", lines 675–679).

was_graceful is decided by unset_signals(), called when pconf is cleared, from the value of is_ungraceful (os/unix/unixd.c lines 506–517). Just below (lines 3664–3671), creating a fresh scoreboard (the children's status board) only when it was not graceful is another difference. The "pipe" used to send children off (POD) and the scoreboard are covered in Part 8.

Easy to get wrong: a graceful restart with a broken new config takes the server down

The first read reporting errors to the shell applies to the newly started process. httpd -k graceful does a first read, so if that process finds an error it prints it, exits with code 1 and never sends the signal. But if you send kill -USR1 to the parent directly, the parent goes straight to the second read inside the loop with no first read. Every failure inside the loop is destroy_and_exit_process(process, 1) (lines 829–831, 838–874). In other words the running parent process exits with code 1. This code path is why the habit of checking the config with httpd -t before a restart matters. (Taking this path by sending kill -USR1 directly was not run in this post's example; I checked it only by following the code path in the source. I also did not check what happens to the children still running at that point.)

6. Where -t, -S and -D DUMP_* stop

Section 1 said -t switches the run mode to AP_SQ_RM_CONFIG_TEST, and -S, -M and -D DUMP_* switch it to AP_SQ_RM_CONFIG_DUMP. These options end at lines 766–777 from section 2. In summary:

The two paths of -V are interesting. When the MPM is built as a .so, as in this series' build, even httpd -V has to read the config file before it can print "Server MPM: event". The comment at line 734 says it is "deferred due to dynamically loaded MPM".

The table's "Where it stops" column laid out top to bottom in execution order. The red boxes are each option's exit.

The dumps are printed not by main() but by the modules hooked into test_config. With -D DUMP_RUN_CFG, the core prints ServerRoot, DocumentRoot and so on (server/core.c lines 5917–5935), and mod_unixd prints User and Group (modules/arch/unix/mod_unixd.c lines 343–361). DUMP_VHOSTS is handled by server/vhost.c and DUMP_MODULES by mod_so. Here is step 5 of the example.

### 5) httpd -t -D DUMP_RUN_CFG
ServerRoot: "/opt/httpd"
Main DocumentRoot: "/opt/httpd/htdocs"
Main ErrorLog: "/tmp/error_log"
Mutex default: dir="/opt/httpd/logs/" mechanism=default 
PidFile: "/tmp/u05.pid"
Define: DUMP_RUN_CFG
User: name="daemon" id=1
Group: name="daemon" id=1
exit status: 0
### /tmp/startlog.txt after -t
pre_config  pid=343   ppid=1     uid=0 state=CREATE_PRE_CONFIG gen=0 static=1 retained=1
(pconf cleanup) pid=343   ppid=1     uid=0 state=other             gen=0 static=2 retained=2

In the first read, the -t family gets only as far as pre_config, check_config and test_config; open_logs and post_config are not called (the example module only logs from pre_config among these, so the log above shows just one line). That is why checking the config does not open log files or create shared memory. The Define: DUMP_RUN_CFG line shows that a name passed with -D goes straight into the list of defines (ap_server_config_defines) that <IfDefine> uses (lines 607–609).

7. Starting as root, dropping to User

A web server usually needs root to open port 80, but handling requests as root is dangerous. So httpd uses the approach of the parent staying root while only the children drop to User. The ps output that run.sh printed in step 1 of the example shows exactly that (parent 154 is root, the children are daemon).

### 1) httpd -k start
pidfile: 154
    PID    PPID USER     COMMAND
    154       1 root     httpd -f /tmp/u05.conf -k start
    155     154 daemon   httpd -f /tmp/u05.conf -k start
    156     154 daemon   httpd -f /tmp/u05.conf -k start

In the log, too, the first and second reads ran as uid=0 and child_init as uid=1.

The work is done not in main.c but in mod_unixd. It is the module that owns the User and Group directives, and it hooks into three places.

static void unixd_hooks(apr_pool_t *pool)
{
    ap_hook_pre_config(unixd_pre_config,
                       NULL, NULL, APR_HOOK_FIRST);
    ap_hook_test_config(unixd_dump_config,
                        NULL, NULL, APR_HOOK_FIRST);
    ap_hook_drop_privileges(unixd_drop_privileges,
                            NULL, NULL, APR_HOOK_MIDDLE);
}

modules/arch/unix/mod_unixd.c lines 363–371

• pre_config: resets User and Group to the defaults chosen at build time (lines 307–340). The config is read afresh every generation, so the previous generation's values must not linger.
• test_config: prints the User: name="daemon" id=1 line from section 6.
• drop_privileges: actually gives up the privileges.

The MPM calls the drop_privileges hook inside the child process. Here is the event MPM's child initialisation code.

    /*stuff to do before we switch id's, so we have permissions. */
    ap_reopen_scoreboard(pchild, NULL, 0);

    /* done with init critical section */
    if (ap_run_drop_privileges(pchild, ap_server_conf)) {
        clean_child_exit(APEXIT_CHILDFATAL);
    }
    /* ... */
    ap_run_child_init(pchild, ap_server_conf);

server/mpm/event/event.c lines 3003–3027

The order matters. As the comment says, it first does the "stuff to do before we switch id's, so we have permissions" (reopening the scoreboard), then gives up privileges, and only then calls the child_init hook. That order is why, in section 3's "Where I got stuck", child_init could not write to the file root had created.

mod_unixd's drop_privileges calls ap_unixd_setup_child(), whose core is setgid() → initgroups() → (chroot() if configured) → setuid() (lines 90–138, lines 146–216). Here is just the user ID part.

PIDs and uids come from the ps output and log of step 1 of the example. Up to the red boxes is root; the green box is daemon.

    /* Only try to switch if we're running as root */
    if (!geteuid() && (
#ifdef _OSD_POSIX
        os_init_job_environment(NULL, ap_unixd_config.user_name, ap_exists_config_define("DEBUG")) != 0 ||
#endif
        setuid(ap_unixd_config.user_id) == -1)) {
        rv = errno;
        ap_log_error(APLOG_MARK, APLOG_ALERT, errno, NULL, APLOGNO(02162)
                    "setuid: unable to change to uid: %ld",
                    (long) ap_unixd_config.user_id);
        return rv;
    }

modules/arch/unix/mod_unixd.c lines 177–188

It calls setuid() only when geteuid() is 0, that is, when it is currently root. If you start httpd as an ordinary user, this part is skipped and it keeps running as that user. That is also why the section 6 dump appends not_used to the User line when not running as root (lines 351–352). The reason setgid() is called before initgroups() is in the comment (lines 116–119): on some platforms setgid() wipes the supplementary group list. setuid() comes last because once root is given up there is no permission left to change groups either — but that is a general Unix rule, not something this file's comments say.

Easy to get wrong: User root is rejected while the config is processed

What happens if you write User root? The User directive's handler returns an error string when the user ID is 0.

    ap_unixd_config.user_name = arg;
    ap_unixd_config.user_id = ap_uname2id(arg);
#if !defined (BIG_SECURITY_HOLE) && !defined (OS2)
    if (ap_unixd_config.user_id == 0) {
        return "Error:\tApache has not been designed to serve pages while\n"
                "\trunning as root.  There are known race conditions that\n"
                "\twill allow any local user to read any file on the system.\n"
                "\tIf you still desire to serve pages as root then\n"
                "\tadd -DBIG_SECURITY_HOLE to the CFLAGS env variable\n"
                "\tand then rebuild the server.\n"
                "\tIt is strongly suggested that you instead modify the User\n"
                "\tdirective in your httpd.conf file to list a non-root\n"
                "\tuser.\n";
    }
#endif

modules/arch/unix/mod_unixd.c lines 228–242

"Apache has not been designed to serve pages while running as root. There are known race conditions that will allow any local user to read any file on the system." This check can only be turned off by defining a macro named BIG_SECURITY_HOLE and rebuilding. #if !defined (...) is preprocessor syntax meaning "include this code if the name is not defined" (see Part 2). The error is raised while directives are being run (step 3 of section 2), so it is caught in the first read and shown in the shell. (I did not actually run with User root; this was checked by following the code path.)

What this post leaves out

To keep to the main thread, the following are only named here.

• How listening sockets (Listen) are kept across restarts — server/listen.c, and ap_setup_listeners() in event_open_logs() (server/mpm/event/event.c lines 3947–3969). Covered in Part 8.
• -k graceful-stop and GracefulShutdownTimeout — lines 3809–3869 of event_run().
• abort_on_oom(), called when memory runs out (lines 310–315), and random number setup in ap_init_rng() (line 554).
• Why apr_hook_sort_all(), which re-sorts the hooks, is called again and again (at the end of ap_setup_prelinked_modules(), server/config.c line 813, and lines 739–740 and 753–758 of the first read) — Part 6.
• ap_server_pre_read_config and ap_server_post_read_config, which let -C and -c add directives before and after the config file (lines 593–601).
• ap_run_optional_fn_retrieve(), which lets optional functions be looked up again (line 879), and apr_pool_lock(), which locks pconf while the MPM runs (lines 877, 884).
• mod_privileges (Solaris), which keeps only part of root's privileges, and mod_systemd, which tells systemd the server is ready — modules/arch/unix/.
• Windows service mode (-k install and friends, server/mpm/winnt/).
• Differences from 2.4.x: the differences I found in main.c are small. On 2.4.x the -D, -C and -c values go into a separate pool called pcommands, whereas trunk uses the process pool directly (line 566). The code that prints "abnormal exit" when the exit code is non-zero and stderr is a terminal (lines 304–306) exists only in trunk. The two-read structure, the loop, and the event MPM becoming a daemon in the second pre_config are the same on the 2.4.x branch.

Summary

What's next

Several times in this post I only said "a module hooks in", "hooks are registered again" or "hooks are sorted" and moved on. How a hook like pre_config is declared in C, who goes first when several modules hook into the same spot, and how LoadModule plugs a .so into the server continue in Part 6: Modules and Hooks — How a Module Plugs into the Server.

Compared with Spring: Spring organises the startup of an application context as twelve steps of a single method, refresh() (Spring Part 7). In httpd the body of main()'s loop plays that role, with the difference that it handles restarts with the same code by reading the config file from scratch and building a new generation. Modules slotting into stages like post_config resembles the way Spring slots post-processors into the bean creation flow (Spring Part 6).

• Part 1: How I Read It with AI, and the Map
• Part 4: APR Memory Pools — Managing Memory Without free()
• Part 7: The Config File and Directives — How One Line of httpd.conf Becomes a C Struct
• Part 8: MPMs — How prefork, worker and event Share Out Connections