Digging into Apache httpd's Source 2: Before Reading a Server Written in C — Structs, Function Pointers, Macros

What this post covers

This is the second post in the "Digging into Apache httpd's Source" series. In Part 1 we fetched and built the source and drew a map of its folders. Before we look at how the server is put together, this time we look at the language httpd is written in: C itself.

This post started from a question asked while preparing the series: "I once read somewhere that C structs are more beautiful than objects in C++ or Java. What are these strengths that only C has, and in what parts of the implementation did they actually pay off?" I don't know which article that was. Remarks like it do come up now and then among people who have written C for a long time. Let me say this up front: "beautiful" is a judgement, not a fact. What this post can do is show, with httpd's source and real runs, what C structs actually give you and what they don't. Whether that is beautiful is for you to decide at the end.

There are three questions to answer.

• What does a C struct look like in memory, and how is that different from a Java or C++ object?
• C has no classes and no interfaces. How did httpd build "modules", the parts you plug in?
• Where does that approach pay off, and what does it cost?

Every line number in this post refers to that commit. Part 1 explains how to get the source at the same commit. This post covers only the main thread — just enough that someone who learned C at school won't get stuck when they open httpd's source.

Words to know first

A module is a part that adds a feature to httpd. You have probably seen lines like LoadModule autoindex_module modules/mod_autoindex.so in httpd.conf; each such line loads one module. httpd builds even its core features as modules. The star of this post is module, the C struct that represents one module.

A directive is one line of the config file, such as Listen 80 or DocumentRoot …. Which module handles which directive is written in a table the module owns (covered in detail in Part 7).

A hook is a spot where you say "at this step of request processing, call my function too". Modules plug into the server by registering functions on hooks (covered in detail in Part 6).

DSOs and .so files. A .so is a library file compiled separately from the server executable and attached while the program runs (the same idea as a .dll on Windows). That is what httpd's LoadModule loads, and httpd calls this feature DSO (Dynamic Shared Object).

An ABI (Application Binary Interface) is "the agreement that compiled machine code must keep with other compiled machine code". If an agreement between pieces of source code (an API) says "this function takes these arguments", an ABI says things at the memory level, like "the third field of this struct sits at byte 8". To attach a separately compiled .so, this agreement has to match.

The bits of C syntax that keep showing up in this post and in httpd's source (structs and ->, function pointers, void *, static, #define, #ifdef) get a short refresher in section 1, using httpd's real code.

1. A quick C refresher — in the shapes httpd uses

If you learned C at school you know these, but in a big C code base they show up in unfamiliar shapes. Here they are, one by one, as you will actually meet them in httpd.

Structs and ->. A struct (struct) bundles several values into one unit. When you have a pointer to a struct, you reach a field with -> instead of .; p->x is short for (*p).x. Most httpd functions that deal with a request take r, a pointer to request_rec (the struct that represents one request), so you see r->handler, r->filename and the like all over the source.

    if (strcmp(r->handler,DIR_MAGIC_TYPE) && !AP_IS_DEFAULT_HANDLER_NAME(r->handler)) {
        return DECLINED;

modules/generators/mod_autoindex.c lines 2303–2304

Function pointers. A function also lives somewhere in memory, so its address can be stored in a variable. Such a variable is a function pointer. The declaration syntax is unusual, and it is where people get stuck most at first.

typedef int (*handler_func)(request_rec *);

server/config.c line 236

Read it from the inside out. (*handler_func) — handler_func is a pointer; (request_rec *) after it — it points to a function taking one request_rec *; int in front — that function returns an int. Because of the typedef, this line does not create a variable; it gives that type the name handler_func. Drop the parentheses (*…) and write int *handler_func(request_rec *), and it means something completely different: "a function returning int *". To call through a function pointer f, both f(r) and (*f)(r) work. httpd uses both.

The reading order from the paragraph above, numbered. The red box is what you get if you leave out one pair of parentheses.

void *. "A pointer whose target type is not stated". Any pointer can be stored in a void *, and you convert (cast) it back to the original type when you use it. In Java terms it is a bit like storing something as Object and casting later, except that nobody checks that the type is right when you take it out. A function that creates a module's configuration returns its own struct as a void *, like this:

static void *create_autoindex_config(apr_pool_t *p, char *dummy)
{
    autoindex_config_rec *new =
    (autoindex_config_rec *) apr_pcalloc(p, sizeof(autoindex_config_rec));
    /* ... */
    return (void *) new;
}

modules/generators/mod_autoindex.c lines 624–646

static. static in front of a function, or in front of a variable at the top level of a file, means "visible only inside this file". It shares a name with Java's static but means something else; it is closer to Java's private. In httpd's source a static function is internal to its file, while functions the core offers to other files and to modules usually carry AP_DECLARE(...). Even the variable that counts modules is hidden inside its file:

static int total_modules = 0;

server/config.c line 206

#define macros. A rule that replaces text before compilation. The stage that does this is called the preprocessor. Some macros take arguments like functions do, but the important point is that they are text substitution, not function calls. From section 3 on, there are a lot of macros.

#ifdef and #ifndef. These include code in, or leave it out of, the compilation depending on whether some name has been #defined. Below is a common pattern: "if nobody set this beforehand, set it to 256".

#ifndef DYNAMIC_MODULE_LIMIT
#define DYNAMIC_MODULE_LIMIT 256
#endif

include/httpd.h lines 119–121

There are bigger examples too. In mod_so.c, which loads .so files, most of the file sits inside #ifndef NO_DLOPEN (line 122), and the #else part (lines 421–438) holds a load_module() with the same name that only logs "LoadModule not supported on this platform". Build on a platform without dynamic loading, and the second one is compiled. When you read the source and wonder "why is this function defined twice?", check the surrounding #if first.

Easy to get confused

• Macros are not functions, so their definitions are hard to find with a debugger or a text search. The ap_run_handler() we'll see in section 6 has no definition anywhere in the source that starts with ap_run_handler(, because a macro builds that name by gluing pieces together.
• The same name can exist as both a macro and a function. ap_get_module_config in section 5 is one.

2. Nothing is hidden in a struct

When people say "C structs are beautiful", the reason most often given is that what you see is all there is. Write two ints in a struct and memory holds two ints. There is no per-object management header and no pointer slipped in behind your back.

Let's check. Below is part of a short C program written for this post (the rest of it continues in section 4).

/* Example written for this post (not httpd source) */
struct point { int x; int y; };
/* ... */
    printf("sizeof(struct point) = %zu, offsetof(struct point, y) = %zu\n",
           sizeof(struct point), offsetof(struct point, y));

sizeof gives a size in bytes, and offsetof(struct, field) gives how many bytes from the start of the struct the field sits. I also made the same shape in C++, but added one virtual function (a function a subclass may replace; in C++ it is marked virtual) to one of them.

/* Example written for this post (not httpd source) */
struct Plain       { int x; int y; };                              /* same shape as the C struct */
struct WithVirtual { int x; int y; virtual ~WithVirtual() {} };    /* has one virtual function */
/* ... main() prints the two sizeof values ... */

These examples (and the module examples further down) were run in a Docker container (Ubuntu 24.04, x86-64, gcc 13.3) with httpd 2.5.1-dev built unchanged from the reference commit (e4af30fe6d).

sizeof(struct point) = 8, offsetof(struct point, y) = 4
C++ Plain = 8 bytes, WithVirtual = 16 bytes

The C struct is exactly what was written, 4 + 4 = 8 bytes, with y at byte 4. In C++, without a virtual function it is also 8 bytes, but adding one virtual function made it 16. The extra 8 bytes are a field we never wrote. It is usually explained as a hidden pointer (the vptr) to a virtual function table (the vtable), but that layout is up to the compiler, and the only thing this post looked at directly is the size. Java objects are also known to carry a header the JVM uses for bookkeeping, but its size and shape are up to the JVM, and this post did not check it.

httpd's module struct is just as plain

I did the same thing with httpd's module struct, module. When the example module mod_u02 used in section 5 receives a request, it prints the size of module and the position of each field (the code is in section 5).

sizeof(module) = 112
  offsetof(version             ) =   0
  offsetof(minor_version       ) =   4
  offsetof(module_index        ) =   8
  offsetof(name                ) =  16
  offsetof(dynamic_load_handle ) =  24
  offsetof(next                ) =  32
  offsetof(magic               ) =  40
  offsetof(rewrite_args        ) =  48
  offsetof(create_dir_config   ) =  56
  offsetof(merge_dir_config    ) =  64
  offsetof(create_server_config) =  72
  offsetof(merge_server_config ) =  80
  offsetof(cmds                ) =  88
  offsetof(register_hooks      ) =  96
  offsetof(flags               ) = 104

Yellow boxes are information the server manages; blue boxes are the functions and tables the module provides. The byte numbers are taken straight from the run.

The first field, version, starts at byte 0. Nothing is hidden in front of it. But "only what you wrote" does not mean "no gaps". module_index (an int, 4 bytes) starts at byte 8 and occupies bytes 8–11, yet the next field, name, starts at byte 16. A pointer is read fastest when it sits at a multiple of 8, so the compiler left 4 bytes empty (alignment). There are also 4 empty bytes after the last field, flags, which brings the size to 112. Those gaps are set by the platform's rules, not by the C standard. There are no hidden fields, but the exact byte positions are only the same "on the same platform, under the same compiler rules".

Why this is useful

Because the memory layout is exactly what the source says, two separately compiled pieces that share the same header file will read the same struct the same way. That is why the httpd executable and a mod_autoindex.so compiled later can hand each other a module. Section 7 shows the checks httpd does so that this agreement doesn't break.

3. A struct of function pointers — a hand-made interface

C has no classes and no interfaces, yet httpd treats dozens of modules the same way. The trick is a struct whose fields are function pointers. Here is the module struct with only its doc comments trimmed:

typedef struct module_struct module;
struct module_struct {
    /* ... */
    int version;
    /* ... */
    int minor_version;
    /** Index to this modules structures in config vectors.  */
    int module_index;

    /** The name of the module's C file */
    const char *name;
    /** The handle for the DSO.  Internal use only */
    void *dynamic_load_handle;

    /* ... */
    struct module_struct *next;

    /* ... */
    unsigned long magic;

    /* ... */
    void (*rewrite_args) (process_rec *process);
    /* ... */
    void *(*create_dir_config) (apr_pool_t *p, char *dir);
    /* ... */
    void *(*merge_dir_config) (apr_pool_t *p, void *base_conf, void *new_conf);
    /* ... */
    void *(*create_server_config) (apr_pool_t *p, server_rec *s);
    /* ... */
    void *(*merge_server_config) (apr_pool_t *p, void *base_conf,
                                  void *new_conf);

    /** A command_rec table that describes all of the directives this module
     * defines. */
    const command_rec *cmds;

    /* ... */
    void (*register_hooks) (apr_pool_t *p);

    /** A bitmask of AP_MODULE_FLAG_* */
    int flags;
};

include/http_config.h lines 347–424

The comment right above the struct reads: "Module structures. Just about everything is dispatched through these" (lines 343–346).

The fields fall into two groups.

• The front (version to rewrite_args): information the server uses to manage the module — API version, the module's slot number in config arrays (module_index), its name, next pointing to the next module, and magic to confirm it really is a module.
• The back (create_dir_config to register_hooks): functions and tables the module hands to the server — "create the directory config with this function", "my directives are in this table", "this function registers my hooks".

In Java, the back half would be an interface with five methods (plus the directive table). The difference is that a method you don't implement can be left as NULL, and the server checks for NULL itself before calling.

    for (modp = ap_top_module; modp; modp = modp->next) {
        if (modp->create_server_config)
            conf_vector[modp->module_index] = (*modp->create_server_config)(p, s);
    }

server/config.c lines 316–319

It walks every module through next and calls the server-config function only for modules that have one (not NULL). Where a Java interface would make you implement every method or provide a default implementation (a default method), C handles it with the convention "if it's empty, skip it".

This is how a real module fills it in

Here is the end of mod_autoindex, the module that shows directory listings:

static void register_hooks(apr_pool_t *p)
{
    ap_hook_handler(handle_autoindex,NULL,NULL,APR_HOOK_MIDDLE);
}

AP_DECLARE_MODULE(autoindex) =
{
    STANDARD20_MODULE_STUFF,
    create_autoindex_config,    /* dir config creater */
    merge_autoindex_configs,    /* dir merger --- default is to override */
    NULL,                       /* server config */
    NULL,                       /* merge server config */
    autoindex_cmds,             /* command apr_table_t */
    register_hooks              /* register hooks */
};

modules/generators/mod_autoindex.c lines 2352–2366

One struct variable is initialized by writing values inside braces, in order. Writing a function's name gives that function's address. The two directory-config functions are there, the two server-config functions are NULL, and then come the directive table autoindex_cmds and the hook-registration function register_hooks. flags is not written at all; in C, when an initializer list is shorter than the struct, the remaining fields are filled with 0.

Two macros fill the first eight slots:

#define APLOG_USE_MODULE(foo) \
    extern module AP_MODULE_DECLARE_DATA foo##_module;                  \
    AP_MAYBE_UNUSED(static int * const aplog_module_index) = &(foo##_module.module_index)
/* ... */
#define AP_DECLARE_MODULE(foo) \
    APLOG_USE_MODULE(foo);                         \
    module AP_MODULE_DECLARE_DATA foo##_module
/* ... */
#define STANDARD20_MODULE_STUFF MODULE_MAGIC_NUMBER_MAJOR, \
                                MODULE_MAGIC_NUMBER_MINOR, \
                                -1, \
                                __FILE__, \
                                NULL, \
                                NULL, \
                                MODULE_MAGIC_COOKIE, \
                                NULL      /* rewrite args spot */

include/http_config.h lines 456–458, lines 467–469, lines 489–496

## is the operator that glues text together inside a macro. AP_DECLARE_MODULE(autoindex) becomes autoindex##_module and defines a variable named autoindex_module. The name you write in LoadModule autoindex_module … in httpd.conf is exactly this variable's name. The aplog_module_index placed before it is for per-module LogLevel, which we'll see in Part 16.

STANDARD20_MODULE_STUFF is nothing more than the values for the first eight fields separated by commas: two API version numbers, -1 for module_index (not decided yet), __FILE__ for the name (a special name the compiler replaces with the current file name), and MODULE_MAGIC_COOKIE for magic.

Seeing the macros expanded

You can see what the macros actually turn into with gcc -E (run the preprocessor and stop). The end of the example module mod_u02.c from section 5 has the same shape as autoindex:

AP_DECLARE_MODULE(u02) = {
    STANDARD20_MODULE_STUFF,
    NULL,                    /* create dir config — none */
    NULL,                    /* merge dir config — none */
    create_server_conf,      /* create server config */
    NULL,                    /* merge server config */
    u02_cmds,                /* directive table */
    register_hooks           /* register hooks */
};

Expanding it in the container with gcc -E -P and turning newlines into spaces gives this:

extern module u02_module; static int * const aplog_module_index __attribute__((unused)) = &(u02_module.module_index); module u02_module = { 20211221, 35, -1, "mod_u02.c", ((void *)0), ((void *)0), 0x41503235UL, ((void *)0), ((void *)0), ((void *)0), create_server_conf, ((void *)0), u02_cmds, register_hooks }

STANDARD20_MODULE_STUFF became the eight values 20211221, 35, -1, "mod_u02.c", NULL, NULL, 0x41503235UL, NULL (((void *)0) is NULL), followed by the six values we wrote. What the compiler sees in the end is a plain initializer with fourteen values in order.

Easy to get confused: positional initializers compile even in the wrong order

With an initializer that lists values by position, as autoindex does, a person has to make sure each value lands in the right field. That is why httpd's modules put a comment such as /* dir config creater */ on every line. If the order is wrong but the types match, the compiler says nothing. I checked:

/* Example written for this post (not httpd source) */
typedef struct ops {
    const char *name;
    int (*on_open)(void);
    int (*on_close)(void);
} ops;

static int do_open(void)  { printf("  -> do_open() ran\n");  return 1; }
static int do_close(void) { printf("  -> do_close() ran\n"); return 0; }

static const ops wrong = { "positional", do_close, do_open };   /* the two functions written in swapped order */
static const ops right = { .name = "designated", .on_open = do_open, .on_close = do_close };
/* ... */
    printf("%s: calling on_open()\n", wrong.name);
    wrong.on_open();
    printf("%s: calling on_open()\n", right.name);
    right.on_open();

Compiled with gcc -Wall -Wextra (options that turn on lots of warnings):

gcc -Wall -Wextra warnings: 0 lines
positional: calling on_open()
  -> do_close() ran
designated: calling on_open()
  -> do_open() ran

Written in positional order, the second value simply lands in the second field; nobody checks whether the names match. The result lines are exactly the output above.

Zero lines of warnings, and calling on_open() ran do_close(). Both functions have the same type, so the compiler has no way to know it is wrong. A designated initializer (C99 and later), which names each field as in .on_open = do_open, prevents this mistake. In httpd's module, merge_dir_config and merge_server_config have exactly the same type (void *(*)(apr_pool_t *, void *, void *)), so swapping those two lines would do the same thing. I could not find in the source why httpd's modules still use positional initializers. Compatibility with old compilers (C89) seems likely, but I found no place that says so, so treat that as a guess. There is something that backs the guess up, though: httpd's headers use designated initializers only when the compiler is "GCC (but not G++), or any other compiler which claims C99 support", switched by AP_HAVE_DESIGNATED_INITIALIZER (include/ap_config.h lines 218–223, include/http_config.h line 76). In other words, httpd still keeps compilers without designated initializers in mind.

4. Const tables of function pointers — how to make a "kind"

The module of section 3 is one struct per module. Now look at the same technique pushed one step further: build a table that holds only function pointers, make it a constant, and have many values point at it. Then "which table does it point to" is the value's kind (its type).

When httpd sends a response, the containers that carry the data are called buckets (covered in detail in Part 11). There are many kinds of bucket — data in memory, a file, a marker such as "end of response" — and each kind is read differently. The struct that defines a "kind" is APR-util's apr_bucket_type_t. With the doc comments trimmed:

struct apr_bucket_type_t {
    /* ... */
    const char *name;
    /* ... */
    int num_func;
    /* ... */
    enum {
        /** This bucket type represents actual data to send to the client. */
        APR_BUCKET_DATA = 0,
        /** This bucket type represents metadata. */
        APR_BUCKET_METADATA = 1
    } is_metadata;
    /* ... */
    void (*destroy)(void *data);

    /* ... */
    apr_status_t (*read)(apr_bucket *b, const char **str, apr_size_t *len, 
                         apr_read_type_e block);
    
    /* ... */
    apr_status_t (*setaside)(apr_bucket *e, apr_pool_t *pool);

    /* ... */
    apr_status_t (*split)(apr_bucket *e, apr_size_t point);

    /* ... */
    apr_status_t (*copy)(apr_bucket *e, apr_bucket **c);

};

APR-util include/apr_buckets.h lines 131–213

A name, the number of functions, whether it is data or a marker (metadata), and five function pointers. httpd adds a kind of its own on top: the EOC (End Of Connection) bucket, a marker that says "end this connection".

AP_DECLARE_DATA const apr_bucket_type_t ap_bucket_type_eoc = {
    "EOC", 5, APR_BUCKET_METADATA,
    apr_bucket_destroy_noop,
    eoc_bucket_read,
    apr_bucket_setaside_noop,
    apr_bucket_split_notimpl,
    apr_bucket_simple_copy
};

server/eoc_bucket.c lines 48–55

Being const, this table never changes while the program runs, and there is exactly one of it in the whole program. Making an EOC bucket just means putting this table's address into the bucket's type field (lines 28–36, b->type = &ap_bucket_type_eoc;). In the bucket struct, that field looks like this:

struct apr_bucket {
    /** Links to the rest of the brigade */
    APR_RING_ENTRY(apr_bucket) link;
    /** The type of bucket.  */
    const apr_bucket_type_t *type;

APR-util include/apr_buckets.h lines 224–228

Reading a bucket goes through this table. apr_bucket_read() is not a function but a macro, and it expands to one line:

#define apr_bucket_read(e,str,len,block) (e)->type->read(e, str, len, block)

APR-util include/apr_buckets.h line 1089

"Call the read in this bucket's kind table." That is what a C++ virtual call does, written out by hand. In section 2 the C++ object grew by 8 bytes because of a single virtual function; here, the thing in that hidden pointer's place is type — a visible field with a name.

The bucket's type field plays the role of C++'s hidden vtable pointer — except that it is not hidden.

Asking for the kind is just as simple. "Is this bucket an EOC?" is one pointer comparison:

#define AP_BUCKET_IS_EOC(e)         (e->type == &ap_bucket_type_eoc)

include/http_connection.h line 208

The job of Java's instanceof is done with an address comparison, using the fact that there is only one table. The kinds APR-util provides by default each have a table of the same shape too (for example "HEAP", "FILE", "EOS").

A small imitation

I imitated the technique in a 30-line C program — the same file as the program in section 2.

/* Example written for this post (not httpd source) */
struct item;
typedef struct item_type {
    const char *name;
    int (*length)(const struct item *it);
    void (*print)(const struct item *it);
} item_type;

typedef struct item {
    const item_type *type;   /* which table it uses = the kind of this value */
    const char *text;
} item;
/* ... text_length, text_print, eos_length, eos_print omitted ... */
static const item_type type_text = { "TEXT", text_length, text_print };
static const item_type type_eos  = { "EOS",  eos_length,  eos_print };
/* ... */
    item list[] = { { &type_text, "hello" }, { &type_text, "httpd" }, { &type_eos, NULL } };
    /* ... */
    for (i = 0; i < sizeof(list) / sizeof(list[0]); i++) {
        list[i].type->print(&list[i]);                  /* call through the table */
        printf("  length = %d\n", list[i].type->length(&list[i]));
    }

[TEXT] "hello"
  length = 5
[TEXT] "httpd"
  length = 5
[EOS] (metadata, no data)
  length = 0

The loop has no idea what kind each value is. It just calls type->print, and each value gets its own kind's function. This is the polymorphism of object-oriented languages (the same call behaving differently depending on the target), built from nothing but structs and function pointers.

Where the strength comes from

• The table is data. To add a new kind, make one more table. httpd added the EOC kind without touching APR-util. No inheritance hierarchy to design.
• The cost is visible. A bucket carries one extra pointer (type), and a call follows two pointers. You can count what costs how much just by reading the source.
• So is the price. Whether all five functions in a table are filled in, and whether read follows the argument rules, is the programmer's responsibility. The comment says "This function is required to be implemented for all bucket types", but the compiler doesn't enforce it (apr_buckets.h lines 158–161).

5. A void * array and module_index — finding config in one step

Every module's config struct has a different shape. autoindex remembers icon lists; other modules remember other things. The core doesn't know those shapes and doesn't need to. httpd solves this with a single void * array. A server (server_rec) and a request (request_rec) each hold arrays like this:

    /** Options set in config files, etc. */
    struct ap_conf_vector_t *per_dir_config;

include/httpd.h lines 1067–1068 (request_rec) · on the server side, module_config at line 1424

The type ap_conf_vector_t exists only as a name, typedef struct ap_conf_vector_t ap_conf_vector_t; (http_config.h line 512); nowhere in this commit's source is there a body (struct ap_conf_vector_t { … }). It is an opaque type: just a name, so outsiders can't look inside. The code that actually builds one shows what it really is:

static ap_conf_vector_t *create_server_config(apr_pool_t *p, server_rec *s)
{
    void **conf_vector = apr_pcalloc(p, sizeof(void *) * conf_vector_length);
    module *modp;

    for (modp = ap_top_module; modp; modp = modp->next) {
        if (modp->create_server_config)
            conf_vector[modp->module_index] = (*modp->create_server_config)(p, s);
    }

    return (ap_conf_vector_t *)conf_vector;
}

server/config.c lines 311–322

It allocates a zero-filled void * array with as many slots as there can be modules (conf_vector_length), and for each module puts the pointer returned by its config function into slot number module_index. When a module fetches its own config, it uses a one-line macro:

#if !defined(AP_DEBUG)

#define ap_get_module_config(v,m)       \
    (((void **)(v))[(m)->module_index])
#define ap_set_module_config(v,m,val)   \
    ((((void **)(v))[(m)->module_index]) = (val))

#endif /* AP_DEBUG */

include/http_config.h lines 551–558

"Read slot [module number] of the array." No lookup by name, no hash to compute — one array index. autoindex fetches its config this way while handling a request (mod_autoindex.c lines 2312–2313).

Who decides module_index? Modules linked into the executable at build time get numbers from 0 upward at startup (config.c lines 776–778), and the array length is generously set to that count plus DYNAMIC_MODULE_LIMIT (256) plus 1 (lines 780–781). A module loaded with LoadModule sees the -1 that STANDARD20_MODULE_STUFF put there and takes the next number at that moment (lines 575–588; the key line is 587, m->module_index = total_modules++;). It is then inserted at the front of the module list (lines 601–604).

The slot numbers are the values from actually running the example below. Yellow is memory; blue is the side that assigns and reads the numbers.

Example: checking it with a module of our own

I wrote a small module called mod_u02. It takes one directive, U02Label, and keeps the value in its server config. When a request comes in for /u02, it prints the size of module from section 2, the list of loaded modules, and what ap_get_module_config returns. Parts already shown (the module struct, the offsetof printing) are trimmed.

/* Example written for this post (not httpd source) */
module AP_MODULE_DECLARE_DATA u02_module;   /* announce the module struct defined below */

/* This module's server config. Its shape is up to the module — the core only handles it as void * */
typedef struct {
    const char *label;       /* the value given by the U02Label directive */
} u02_server_conf;

static void *create_server_conf(apr_pool_t *p, server_rec *s)
{
    u02_server_conf *conf = apr_pcalloc(p, sizeof(*conf));  /* taken from the config pool — no free */
    conf->label = "(unset)";
    return conf;
}

static const char *set_label(cmd_parms *cmd, void *dummy, const char *arg)
{
    u02_server_conf *conf = ap_get_module_config(cmd->server->module_config, &u02_module);
    conf->label = arg;
    return NULL;
}
/* ... */
    /* 3) the loaded modules — a linked list joined by next pointers */
    ap_rprintf(r, "loaded modules (ap_top_module -> next ...):\n");
    for (m = ap_top_module; m; m = m->next) {
        ap_rprintf(r, "  [%2d] %s\n", m->module_index, m->name);
    }

    /* 4) ap_get_module_config just reads slot [module_index] of the array */
    conf = ap_get_module_config(r->server->module_config, &u02_module);
    by_hand = ((void **)r->server->module_config)[u02_module.module_index];
    ap_rprintf(r, "ap_get_module_config = %pp, by hand = %pp, same = %s, label = %s\n",
               (void *)conf, by_hand, (void *)conf == by_hand ? "yes" : "no", conf->label);

    /* 5) a mistake: look in the directory config array? It compiles silently and gives NULL */
    dir_conf = ap_get_module_config(r->per_dir_config, &u02_module);
    ap_rprintf(r, "per_dir_config slot for u02 = %s\n", dir_conf ? "not NULL" : "NULL");

I compiled it with apxs -c mod_u02.c, started httpd with a config that loads an MPM, a few required modules, mod_autoindex and this module via LoadModule, plus U02Label hello-from-httpd.conf, and sent curl http://127.0.0.1:8080/u02. The size and field offsets of module are copied into section 2; the rest looks like this:

u02_module: version=20211221 minor=35 module_index=7 name=mod_u02.c flags=0
u02_module: magic=0x41503235 ("AP25")
loaded modules (ap_top_module -> next ...):
  [ 7] mod_u02.c
  [ 6] mod_autoindex.c
  [ 5] mod_authz_core.c
  [ 4] mod_unixd.c
  [ 3] event.c
  [ 2] http_core.c
  [ 1] mod_so.c
  [ 0] core.c
ap_get_module_config = 7f81eb6dcc58, by hand = 7f81eb6dcc58, same = yes, label = hello-from-httpd.conf
per_dir_config slot for u02 = NULL

(The pointer address 7f81eb6dcc58 changes from run to run.)

• core.c, mod_so.c and http_core.c are linked into the executable and got 0, 1 and 2; the five loaded with LoadModule got 3 to 7 in config-file order. The MPM (event.c) is a module too. What gets linked into the executable is covered in Part 3.
• In the list, the most recently loaded comes first, because ap_add_module() inserts each new module at the front.
• name is the value __FILE__ put there, but only the file name is left, without a path. ap_add_module() cuts it down to what follows the last / (lines 625–629).
• The result of ap_get_module_config and reading slot [7] of the array by hand are the same address, and it holds the value written in httpd.conf.
• Read the four bytes of magic as characters and you get "AP25". It comes back in section 7.

How Spring solved the same problem

A Spring container also does the job of "give me a name, I'll find you the object". As we saw in Spring Part 4, finished beans live in a ConcurrentHashMap keyed by bean name, and a lookup goes through that hash map using the name string. When you don't know in advance how many beans there will be or what they'll be called, a map is the natural choice. httpd instead imposes the constraint "modules are all settled at startup, and each gets a number", and in return finding a config becomes reading one array slot. It is less a question of which is better than of what you chose to fix in advance.

Easy to get confused

• It is both a macro and a function. Just above in the same header, ap_get_module_config is also declared as a function (http_config.h lines 522–523), and its body is in server/util_debug.c (lines 98–108). In a normal build without AP_DEBUG, the macro inside the #if at line 551 shadows the name, so the macro is what gets used. Expanding it with gcc -E in the container shows it really did become array indexing:

  conf = (((void **)(r->server->module_config))[(&u02_module)->module_index])
  

• void * checks nothing. The last line of the example above shows it. mod_u02 left its directory-config functions as NULL, so slot 7 of the directory config array is empty. If you make the mistake of looking in the directory config array (r->per_dir_config) instead of the server config array (r->server->module_config), the compiler says nothing, and at run time you get NULL. Had the code read ->label through that NULL, the server process would have crashed (the example stops just before that). Casting a void * to the wrong struct type compiles just as happily. Where Java would throw a ClassCastException, C quietly reads the wrong memory.

• The first time I ran the example, the sizes came out as the literal text %zu instead of numbers. ap_rprintf() doesn't use the C standard printf but APR's formatter, and the formatter in APR 1.7.2 has no z modifier for size_t (APR strings/apr_snprintf.c lines 844–858). So the example casts to (int) and prints with %d. APR has a separate macro, APR_SIZE_T_FMT, for exactly this case.

6. Hook functions stamped out by macros

Modules plug into the server by registering functions on hooks. The ap_hook_handler(handle_autoindex, NULL, NULL, APR_HOOK_MIDDLE) called in autoindex's register_hooks() is such a registration. Yet if you search the source for the definition of ap_hook_handler(), you won't find it. That function is produced by a macro.

A hook comes into being with two lines. It is declared in a header,

AP_DECLARE_HOOK(int,handler,(request_rec *r))

include/http_config.h line 1399

and implemented in a .c file:

AP_IMPLEMENT_HOOK_RUN_FIRST(int, handler, (request_rec *r),
                            (r), DECLINED)

server/config.c lines 169–170

httpd's AP_… macros only add a name prefix (ap, AP) and pass everything on to APR-util's macros (include/ap_hooks.h lines 74–75, lines 137–139). The real function bodies are in APR-util's macros. Here is the run-function part of RUN_FIRST:

#define APR_IMPLEMENT_EXTERNAL_HOOK_RUN_FIRST(ns,link,ret,name,args_decl,args_use,decline) \
APR_IMPLEMENT_EXTERNAL_HOOK_BASE(ns,link,name) \
link##_DECLARE(ret) ns##_run_##name args_decl \
    { \
    ns##_LINK_##name##_t *pHook; \
    int n; \
    ret rv = decline; \
    APR_HOOK_INT_DCL_UD; \
\
    APR_HOOK_PROBE_ENTRY(ud, ns, name, args_use); \
\
    if(_hooks.link_##name) \
        { \
        pHook=(ns##_LINK_##name##_t *)_hooks.link_##name->elts; \
        for(n=0 ; n < _hooks.link_##name->nelts ; ++n) \
            { \
            APR_HOOK_PROBE_INVOKE(ud, ns, name, (char *)pHook[n].szName, args_use); \
            rv=pHook[n].pFunc args_use; \
            APR_HOOK_PROBE_COMPLETE(ud, ns, name, (char *)pHook[n].szName, rv, args_use); \
\
            if(rv != decline) \
                break; \
            } \
        } \
\
    APR_HOOK_PROBE_RETURN(ud, ns, name, rv, args_use); \
\
    return rv; \
    }

APR-util include/apr_hooks.h lines 267–295

Put ap in ns, handler in name, (request_rec *r) in args_decl, (r) in args_use and DECLINED in decline, glue the ## pieces, and out comes this function: int ap_run_handler(request_rec *r) — call the registered function pointers (the array _hooks.link_handler) in order, and stop at the first value that is not DECLINED, returning it. The APR_HOOK_PROBE_… lines are slots for tracing tools and expand to nothing in a normal build (apr_hooks.h lines 52–69 and so on, when APR_HOOK_PROBES_ENABLED is not defined). The APR_IMPLEMENT_EXTERNAL_HOOK_BASE on the first line produces the registration function ap_hook_handler(), a function that pushes the function pointer it receives onto that array (lines 143–165).

When request processing reaches the point of picking a handler, the core calls ap_run_handler(r) (server/config.c line 443). autoindex's handle_autoindex() returns DECLINED for anything that isn't a directory request, passing it on to the next module (lines 2303–2304 from section 1); when it matches, it builds the response itself and returns something else, which stops the loop there.

Grey is the macros written in the source and the callers; blue is the functions the macros produce; yellow is the function pointer array.

Write those two lines per hook and you get one registration function and one run function each. Even within the single file server/config.c, hooks like header_parser, pre_config, post_config and handler are made the same way (from line 85). The RUN_ALL variant that "calls everything", the VOID variant with no return value, and how the order of functions on the same hook is decided (APR_HOOK_MIDDLE and friends) are in Part 6.

Easy to get confused

• Macros stand in for the "generics" C doesn't have. Each hook has different argument and return types, and C has no functions that take types as arguments. So text substitution stamps out one set of functions per hook. Thanks to that, ap_run_handler() is a correctly typed function taking a request_rec *. In exchange, the source of that function exists nowhere as text, and first-time readers can lose a long time on "where is this function?". If the name is ap_run_xxx or ap_hook_xxx, look for AP_IMPLEMENT_HOOK_…(…, xxx, …).
• Spring solved the same problem with interfaces and lists. The BeanPostProcessor of Spring Part 6 is an interface with fixed methods, and the container walks the list of registered implementation objects and calls them in turn. The skeleton — "call the registered ones in order" — is the same. Java defines one interface per extension point; httpd uses two macro lines to make a function pointer array and the function that runs it.

7. Trusting a separately compiled .so — ABI and MMN

Section 2 said that "sharing the same header means reading the same shape". Turn that around: with a different header, you read it wrong. And C has no way to notice at run time. The module inside a .so file is just 112 bytes; no "this is the shape I was compiled with" information is attached automatically. httpd guards against this with two numbers written directly into fields.

mod_so, which processes LoadModule, opens the .so, then uses the name given in the directive (u02_module) to find the address of that variable inside it, and treats it as a module *.

    if (apr_dso_sym(&modsym, modhandle, modname) != APR_SUCCESS) {
        /* ... */
    }
    modp = (module*) modsym;
    modp->dynamic_load_handle = (apr_dso_handle_t *)modhandle;
    modi->modp = modp;

    /*
     * Make sure the found module structure is really a module structure
     *
     */
    if (modp->magic != MODULE_MAGIC_COOKIE) {
        return apr_psprintf(cmd->pool, "API module structure '%s' in file %s "
                            "is garbled - expected signature %08lx but saw "
                            "%08lx - perhaps this is not an Apache module DSO, "
                            "or was compiled for a different Apache version?",
                            modname, module_file,
                            MODULE_MAGIC_COOKIE, modp->magic);
    }

modules/core/mod_so.c lines 276–299

(module*) modsym is a cast that takes an address and simply declares "this is a module". From the very next line on, it writes to and reads from that struct. The only way to confirm it is right is to check that magic, at byte 40, holds the agreed value. Once that passes, ap_add_loaded_module() → ap_add_module() checks once more (line 304).

    if (m->version != MODULE_MAGIC_NUMBER_MAJOR) {
        return apr_psprintf(p, "Module \"%s\" is not compatible with this "
                            "version of Apache (found %d, need %d). Please "
                            "contact the vendor for the correct version.",
                            m->name, m->version, MODULE_MAGIC_NUMBER_MAJOR);
    }

server/config.c lines 568–573

Both numbers live in include/ap_mmn.h. MMN stands for Module Magic Number.

#define MODULE_MAGIC_COOKIE 0x41503235UL /* "AP25" */

#ifndef MODULE_MAGIC_NUMBER_MAJOR
#define MODULE_MAGIC_NUMBER_MAJOR 20211221
#endif
#define MODULE_MAGIC_NUMBER_MINOR 35             /* 0...n */

include/ap_mmn.h lines 749–754

The comment at the top of the same file states the rules. MAJOR is bumped for "major API changes that could cause compatibility problems for older modules such as structure size changes", and across a change of MAJOR no binary compatibility is possible. MINOR is bumped for additions that don't break compatibility (lines 24–31). Between that comment and the definitions runs a long log with one line for each bump saying what changed (the last entry at this commit is 20211221.35). On the 2.4.x branch, the one most people install, the same spot reads 0x41503234UL /* "AP24" */ with MAJOR 20120211. So a module compiled for 2.4 won't load as-is into the trunk httpd.

The two red boxes are where it ends in an error. The one triggered on purpose in the example below is "not compatible".

Example: loading a module compiled with an old MMN

The MAJOR definition in ap_mmn.h is wrapped in #ifndef, so a different value can be supplied from outside at compile time. Using that, I compiled the same mod_u02.c with 2.4.x's MAJOR value to imitate "a module for an older release".

apxs -c -DMODULE_MAGIC_NUMBER_MAJOR=20120211 mod_u02.c
httpd -t -f /tmp/u02old.conf

httpd: Syntax error on line 8 of /tmp/u02old.conf: Module "mod_u02.c" is not compatible with this version of Apache (found 20120211, need 20211221). Please contact the vendor for the correct version.

magic is still "AP25", so the first check passes and the second one catches it. The module's version field holds MODULE_MAGIC_NUMBER_MAJOR as it was when the module was compiled, and the server compares it with the value from when the server itself was compiled. That one comparison is what decides "may I read these 112 bytes in the shape I know?". A real 2.4 module would already have "AP24" in magic and would be caught by the first check (garbled). I did not run that case; it follows from the order of the checks in the source above.

Strengths and costs

• C's ABI is simple. A struct lays its fields out in order (the gap rules are the platform's), and functions are exported as symbols under their own names. That is why a .so compiled at another time, by another person, can be found by one name (u02_module) and attached. The whole ecosystem of third-party modules stands on that.
• In exchange, keeping it intact is up to people. Insert a field into module, or reorder the fields of request_rec, and every already-compiled module reads the wrong bytes. httpd manages that through the rule of bumping the MMN. New fields are often appended at the end of a struct (for example module's flags, with AP_MODULE_FLAGS_MMN_MAJOR/MINOR to check for it, http_config.h lines 539–543) so that the earlier fields keep their positions.
• In Java, a .class file carries the names and type information of its class and methods, and the JVM is known to match them up when it loads the class. This post did not verify that against the JVM's source. What can be said for certain is about httpd: in C that information doesn't appear by itself, so httpd writes numbers into the struct and compares them directly.

8. Tying memory lifetimes to pools (a teaser)

One more short one to finish. The example module in section 5 got its config struct from apr_pcalloc(p, …) and never called free. The same was true when the core built the config array (the apr_pcalloc in create_server_config() in section 5). There is no garbage collector (GC) as in Java, so what happens?

p is a pool. Memory taken from a pool is all given back at once when the pool is cleared. And not just memory: after loading a .so, mod_so registers a cleanup function so that the .so is unloaded when that pool is cleared.

    /*
     * Register a cleanup in the config apr_pool_t (normally pconf). When
     * we do a restart (or shutdown) this cleanup will cause the
     * shared object to be unloaded.
     */
    apr_pool_cleanup_register(cmd->pool, modi, unload_module, apr_pool_cleanup_null);

modules/core/mod_so.c lines 309–314

According to the comment, this pool is normally the config pool (pconf), which is cleared on restart or shutdown. The pool passed a few lines later to ap_single_module_configure(), which creates the module's config, is the same cmd->pool (line 319), so the example module's config struct and the .so itself are tied to the same pool. It is a way of declaring "this dies together with this pool". Instead of working out "nobody points to it any more, so clean it up" as a GC does, it ties things to units with a clear lifetime — configuration, connection, request. The cost is the mistakes you make by picking the wrong lifetime. Store the address of memory from a request pool somewhere that lives longer, such as a config struct, and after the request ends that address points to memory that has already been given back — and C does nothing to stop it. What pools look like, and where in httpd they are created and cleared, is covered in detail in Part 4.

The yellow boxes are tied to the same pool and die together (dso_cleanup, which actually closes the .so, is registered at line 162 of APR's dso/unix/dso.c). The red boxes are what happens when you pick the wrong lifetime. Pools themselves are covered in Part 4.

9. Putting it together — what C gave, and what it cost

Here is how Java (Spring) does the same jobs, side by side. The httpd column is what this post checked in the source; the Java column is general language features plus what we saw in the Spring series.

What C gave can be summed up as "what you see is all there is". The shape of a struct, the cost of a call, the cost of finding a config are all written plainly in the source, and that simplicity is what lets a separately compiled part be attached by a single name. If what "beautiful" points to is this transparency, httpd is a good example of it.

The table's "Same call, different behaviour" and "Checking the kind" rows as a picture. The left side is what we read in the source in section 4; the right side is a general feature of the language.

What C won't do for you we also saw first-hand.

• No type checking. Looking up void * config in the wrong array compiles (section 5; you get NULL).
• Positional initializers compile in the wrong order. With zero warnings, do_close ran in the on_open slot (section 3).
• Macros are hard to read. ap_run_handler() doesn't exist as text in the source (section 6).
• Compatibility is kept by people. Change a struct and you must bump the MMN, and the check is just two numbers (section 7).
• Lifetimes are decided by people too. Pick the wrong pool and you end up pointing to memory already given back (section 8).

httpd's source fills these gaps with rules, comments and conventions: the comment on every line of a module initializer, the long change log in ap_mmn.h, sentences like "this function is required to be implemented for all bucket types". That is why such comments matter when we read the source in the rest of this series: they are where the agreements are written down, in place of a compiler.

What this post left out

• The directive table command_rec and macros like AP_INIT_TAKE1 (around include/http_config.h line 128), and merging directory configs with ap_merge_per_dir_configs() (server/config.c lines 285–309) — Part 7.
• Sorting hook functions (apr_hook_sort_all), the RUN_ALL and VOID variants, optional functions — Part 6.
• The rest of buckets and brigades (the ring structure, splitting and copying buckets) — Part 11.
• AP_DEBUG, under which debug builds use the function instead of the ap_get_module_config macro, and ap_get_core_module_config, which fetches the core's config quickly (include/http_core.h lines 381–389).
• ap_remove_module(), which empties the slot when a module is unloaded (server/config.c line 662).

Summary

What's next

Now that we've seen what C gives and what it doesn't, we look at the shape of httpd as a whole: how APR sits on the operating system, the core on APR, the MPM that shares out connections, and the modules, in layers; how even the core is a module called core_module; and the three big structs you'll meet everywhere in the source, server_rec, conn_rec and request_rec. That is Part 3: The Big Picture — Core, APR, MPM, Modules and the Three Big Structs. You'll also see where the module struct and the module_config array from this post sit on that map.

• Part 1: How I Read It with AI, and the Map
• Part 6: Modules and Hooks — How a Module Plugs into the Server
• Spring built its extension points as interfaces: Spring Part 6: Lifecycle Callbacks and PostProcessors
• Creating an object that implements an interface at run time and slotting it in: Spring Part 10: How AOP Proxies Are Made