Digging into nginx's Source 2: The Big Picture — Master and Workers, Modules, the Cycle and Config Contexts

What this post covers

This is the second post in the "Digging into nginx's Source" series. In Part 1 we fetched and built the source, and laid out a folder map and the main road a single request travels. This post is the big-picture map we unfold before reading code line by line. Parts 3 through 12 each pick one box on this map and zoom in.

If you have used nginx, you have seen nginx: master process and nginx: worker process in ps, and you have nested events {}, http {}, server {} and location {} in nginx.conf. Using those familiar sights as a bridge, this post answers:

• Which functions in the source are the master and the workers, and what does each wait for most of the time?
• What does a "module" look like in C, and how do the events, http and stream blocks relate to module types?
• What does ngx_cycle_t, the state of one running nginx, contain, and where does each module's config struct hang?
• Why does putting proxy_pass directly inside http {} give "not allowed here" — what is a config context in the source?

All line numbers in this post refer to that commit. How to fetch the same commit and build the Docker image for the examples is in Part 1. Paths start at the repository root (src/core/…).

Since this post draws a map, it follows only the main lines. Each section points to the post that reads that part in detail ("we'll see it in Part N").

Terms to know first

The master process is the first process left running after you start nginx. It usually runs as root; it reads the config, opens the ports, creates the workers and receives signals (commands like nginx -s reload). It does not handle requests.

A worker process is a child the master creates with fork(). There are as many as worker_processes says, and they accept the actual connections and handle requests. Each worker runs on a single thread (we confirm this in sections 2 and 6).

A module is nginx's unit of functionality. HTTP handling, logging, gzip and proxying are modules — but so are "the core that handles directives at the top level of the config file" and "event handling with epoll". In C a module is one variable of type ngx_module_t (section 3).

The cycle is the struct ngx_cycle_t holding the whole state of one running nginx: the parsed config, the open listening sockets, the connection array and the module list all hang off it. When the config is re-read (reload), nginx builds another cycle — hence the name (section 4).

A configuration context is where a directive sits: the top level of the file (main), events {}, http {}, server {}, location {} and so on. Each directive declares, as bit flags, which contexts it may appear in (section 5).

An event loop is a loop that forever repeats "wake up and handle whatever socket is ready; if nothing is, sleep". On Linux nginx sleeps in epoll (a Linux system call where you register thousands of sockets at once and get back only those that became readable or writable). Sockets are used in non-blocking mode (a read with no data available returns "nothing yet" immediately instead of waiting) (section 6).

1. The map of the whole series

Let's start with one picture. The nginx source is easiest to read in three parts.

The series map. Parts 3–12 refer to "the ○○ box on the Part 2 map" by these box names. Yellow is memory, blue is configuration, green is processes and events, gray is request handling.

• A. Preparation — what nginx does once at startup (and again at reload). main() reads nginx.conf, fills in a config struct for each module and collects the result in one cycle. All of that memory comes from pools (bundles of memory thrown away all at once).
• B. Processes and events — once preparation is done, the master creates workers, and each worker runs an event loop accepting connections.
• C. One HTTP request — read a request from a connection, pass through 11 phases (the ordered stages of request handling — rewriting, access checks, generating the response and so on) while choosing a location, then send the response out through the filter chain (a chain of functions that modify and emit the response headers and body in turn) or proxy it to an upstream (the backend server nginx hands the request to).

The rest of this post walks through five skeletons that span the whole map — processes (section 2), modules (section 3), the cycle (section 4), config contexts (section 5) and the event loop (section 6). Finally we look at how HTTP, stream and mail stand on the same core (section 7).

2. Processes — one master, several workers

The end of main(): which loop do we enter?

After main() reads the config and builds the cycle (Part 4), it ends by entering one of two loops and never coming back.

    if (ngx_process == NGX_PROCESS_SINGLE) {
        ngx_single_process_cycle(cycle);

    } else {
        ngx_master_process_cycle(cycle);
    }

    return 0;
}

src/core/nginx.c lines 399–407

The master_process directive decides which. Just above, if ccf->master is on, ngx_process becomes NGX_PROCESS_MASTER (lines 346–348). The default is on — ngx_conf_init_value(ccf->master, 1); in ngx_core_module_init_conf() (line 1192). With master_process off; nginx goes to ngx_single_process_cycle(), where one process runs the event loop itself. That is handy under gdb, so later posts use it now and then.

The same function holds the default for worker_processes: ngx_conf_init_value(ccf->worker_processes, 1); (line 1196) — leave it out and you get one worker.

The master: create workers, wait for signals

    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);

    ngx_start_worker_processes(cycle, ccf->worker_processes,
                               NGX_PROCESS_RESPAWN);
    ngx_start_cache_manager_processes(cycle, 0);
    /* ... */
    for ( ;; ) {
        /* ... */
        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "sigsuspend");

        sigsuspend(&set);

        ngx_time_update();
        /* ... */
        if (ngx_reconfigure) {
            ngx_reconfigure = 0;
            /* ... */
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reconfiguring");

            cycle = ngx_init_cycle(cycle);
            if (cycle == NULL) {
                cycle = (ngx_cycle_t *) ngx_cycle;
                continue;
            }

            ngx_cycle = cycle;
            ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
                                                   ngx_core_module);
            ngx_start_worker_processes(cycle, ccf->worker_processes,
                                       NGX_PROCESS_JUST_RESPAWN);
            ngx_start_cache_manager_processes(cycle, 1);

            /* allow new processes to start */
            ngx_msleep(100);

            live = 1;
            ngx_signal_worker_processes(cycle,
                                        ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
        }

src/os/unix/ngx_process_cycle.c lines 128–266

That is almost everything the master does.

1. It creates worker_processes workers. ngx_start_worker_processes() calls ngx_spawn_process(cycle, ngx_worker_process_cycle, …, "worker process", type) in a loop (lines 358–371), and ngx_spawn_process() creates a parent–child communication pipe (the channel) with socketpair() and then calls fork(). The child calls the function pointer it was handed, proc — that is, ngx_worker_process_cycle() (src/os/unix/ngx_process.c lines 186–200).
2. It sleeps in sigsuspend() and wakes when a signal arrives.
3. The signal handler ngx_signal_handler() does no real work; it only sets global flags — ngx_reconfigure = 1 for SIGHUP (src/os/unix/ngx_process.c lines 364–365), ngx_quit = 1 for SIGQUIT, ngx_reap = 1 for SIGCHLD (a child exited). The awakened loop looks at the flags and does the work. This is a common pattern, because very little is safe to do inside a signal handler.
4. On reload (ngx_reconfigure) it calls ngx_init_cycle(cycle) to build a new cycle from the old one, starts new workers, and tells the old workers to stop (NGX_SHUTDOWN_SIGNAL, i.e. QUIT).

When you type nginx -s reload, a freshly started nginx process reads the master's pid from the pid file, sends it kill(pid, SIGHUP) and exits (ngx_signal_process() → ngx_os_signal_process(), src/core/ngx_cycle.c lines 1097–1145). The pairing of the name (reload) and the signal (SIGHUP) lives in the signals[] table (src/os/unix/ngx_process.c lines 39–83).

The big steps of one reload. New workers start before old ones exit, so connections in progress are not dropped. Details in Part 6.

To tell workers "stop", the master uses the channel before signals. ngx_signal_worker_processes() turns QUIT into the NGX_CMD_QUIT command and writes it to the channel (src/os/unix/ngx_process_cycle.c line 471, line 519), and only sends a signal with kill() if that write fails (line 534). On the worker side, ngx_channel_handler() reads the command and sets ngx_quit = 1 (lines 1091–1093). The channel is registered in the worker's event loop as just another socket, so the worker learns "the master is talking to me" exactly the way it learns "a connection arrived".

The worker: run the event loop forever

static void
ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data)
{
    ngx_int_t worker = (intptr_t) data;

    ngx_process = NGX_PROCESS_WORKER;
    ngx_worker = worker;

    ngx_worker_process_init(cycle, worker);

    ngx_setproctitle("worker process");

    for ( ;; ) {

        if (ngx_exiting) {
            if (ngx_event_no_timers_left() == NGX_OK) {
                ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
                ngx_worker_process_exit(cycle);
            }
        }

        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "worker cycle");

        ngx_process_events_and_timers(cycle);

        if (ngx_terminate) {
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
            ngx_worker_process_exit(cycle);
        }

        if (ngx_quit) {
            ngx_quit = 0;
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0,
                          "gracefully shutting down");
            ngx_setproctitle("worker process is shutting down");

            if (!ngx_exiting) {
                ngx_exiting = 1;
                ngx_set_shutdown_timer(cycle);
                ngx_close_listening_sockets(cycle);
                ngx_close_idle_connections(cycle);
                ngx_event_process_posted(cycle, &ngx_posted_events);
            }
        }
        /* ... */
    }
}

src/os/unix/ngx_process_cycle.c lines 728–779

The worker's body is a loop that calls ngx_process_events_and_timers() forever. That one line is one turn of the event loop in section 6. Between turns it checks flags with the same names as the master's (ngx_terminate, ngx_quit, ngx_reopen). On ngx_quit it does not exit at once: it sets ngx_exiting = 1, closes the listening sockets and idle connections, and keeps running until nothing is left (ngx_event_no_timers_left()). That is the "gracefully shutting down" you see in the log.

Before the loop, ngx_worker_process_init() drops privileges to the user directive's user (which is why workers run as nobody) and calls every module's init_process callback (lines 934–940). The event module's init_process, ngx_event_process_init(), is where the connection array is built and the listening sockets are registered with epoll (section 6).

Example 1: seeing both loops with ps and gdb

All examples in this post were run with nginx 1.31.9 built as-is from the base commit (9a7439fd79) in a Docker container (Debian bookworm, --with-debug). There is one script, ex/u02-architecture/run.sh; the === lines in the output are headings it prints. The config is the smallest one with two workers and one each of events, http, server and location.

# Example written for this post (not part of the nginx source)
# Part 2: one master + two workers, the smallest config with one events, http, server and location context each
worker_processes 2;
error_log /tmp/u02/error.log notice;
pid /tmp/u02/nginx.pid;

events {
    worker_connections 1024;
}

http {
    access_log off;

    server {
        listen 8080;

        location = /hello {
            return 200 "hello from $nginx_version\n";
        }

        location = /status {
            stub_status;
        }
    }
}

After starting nginx, I looked at the processes with ps and attached gdb to the master and a worker to run bt (prints the chain of calls that led to where the process is currently stopped, innermost first).

nginx -c /work/nginx.conf
ps -o pid,ppid,user,nlwp,cmd -C nginx
gdb -batch -p "$MASTER" -ex 'bt'
gdb -batch -p "$WORKER" -ex 'bt'

=== 3. start: one master, two workers ===
hello from 1.31.9
    PID    PPID USER     NLWP CMD
     45       1 root        1 nginx: master process nginx -c /work/nginx.conf
     46      45 nobody      1 nginx: worker process
     47      45 nobody      1 nginx: worker process
...
=== 5. where is each process waiting? (gdb bt) ===
--- master (pid 45) ---
#0  0x00007f94ed8482d5 in sigsuspend () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055a5987780be in ngx_master_process_cycle (cycle=0x55a599b7d120) at src/os/unix/ngx_process_cycle.c:163
#2  0x000055a59872d1c0 in main (argc=3, argv=0x7ffdcc903668) at src/core/nginx.c:403
--- worker (pid 46) ---
#0  0x00007f94ed914ef3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055a59877c608 in ngx_epoll_process_events (cycle=0x55a599b7d120, timer=18446744073709551615, flags=1) at src/event/modules/ngx_epoll_module.c:800
#2  0x000055a5987690b9 in ngx_process_events_and_timers (cycle=0x55a599b7d120) at src/event/ngx_event.c:248
#3  0x000055a598779a4c in ngx_worker_process_cycle (cycle=0x55a599b7d120, data=0x0) at src/os/unix/ngx_process_cycle.c:751
#4  0x000055a598776148 in ngx_spawn_process (cycle=0x55a599b7d120, proc=0x55a598779956 <ngx_worker_process_cycle>, data=0x0, name=0x55a59889af0a "worker process", respawn=-3) at src/os/unix/ngx_process.c:199
#5  0x000055a598778826 in ngx_start_worker_processes (cycle=0x55a599b7d120, n=2, type=-3) at src/os/unix/ngx_process_cycle.c:366
#6  0x000055a598777eef in ngx_master_process_cycle (cycle=0x55a599b7d120) at src/os/unix/ngx_process_cycle.c:130
#7  0x000055a59872d1c0 in main (argc=3, argv=0x7ffdcc903668) at src/core/nginx.c:403

PIDs and addresses differ from run to run.

• The master runs as root, the two workers as nobody. NLWP (number of threads in the process) is 1 for all three. A worker runs on one thread.
• The master is asleep at line 163 of ngx_process_cycle.c — the sigsuspend(&set); in the excerpt above.
• Read the worker's stack bottom-up and you get this section: main → ngx_master_process_cycle (line 130, which starts the workers) → ngx_start_worker_processes → ngx_spawn_process (line 199, proc(cycle, data)) → ngx_worker_process_cycle (line 751, ngx_process_events_and_timers(cycle)) → ngx_epoll_process_events → epoll_wait. Because the worker was fork()ed inside the master's function, the master's frames (#5–#7) remain on the worker's stack too.
• timer=18446744073709551615 is (ngx_msec_t) -1, i.e. NGX_TIMER_INFINITE (src/event/ngx_event_timer.h line 17). No timer is pending, so it waits indefinitely for a connection. flags=1 is NGX_UPDATE_TIME.

Easy to confuse: the master and worker have the same cycle address

In the output, the master's cycle=0x55a599b7d120 and the worker's cycle=0x55a599b7d120 are identical. They are not sharing memory. fork() copies the parent's whole address space, so the child simply has a cycle with the same contents at the same address; after that, changes on one side are invisible to the other. Values that workers really do share (upstream_zone, limit_req_zone, accept_mutex and so on) live separately in shared memory (a taste in Part 3, upstream_zone in Part 12).

3. Everything is a module

One module = one ngx_module_t variable

In nginx the unit for adding functionality is the module, and in C a module is this one struct.

struct ngx_module_s {
    ngx_uint_t            ctx_index;
    ngx_uint_t            index;

    char                 *name;

    ngx_uint_t            spare0;
    ngx_uint_t            spare1;

    ngx_uint_t            version;
    const char           *signature;

    void                 *ctx;
    ngx_command_t        *commands;
    ngx_uint_t            type;

    ngx_int_t           (*init_master)(ngx_log_t *log);

    ngx_int_t           (*init_module)(ngx_cycle_t *cycle);

    ngx_int_t           (*init_process)(ngx_cycle_t *cycle);
    ngx_int_t           (*init_thread)(ngx_cycle_t *cycle);
    void                (*exit_thread)(ngx_cycle_t *cycle);
    void                (*exit_process)(ngx_cycle_t *cycle);

    void                (*exit_master)(ngx_cycle_t *cycle);

    uintptr_t             spare_hook0;
    /* ... */
};

src/core/ngx_module.h lines 223–258

It reads more easily in three groups.

The key is that ctx is a void *. For a CORE module it points to an ngx_core_module_t (a name, create_conf, init_conf) (src/core/ngx_module.h lines 261–265); for an HTTP module it points to an ngx_http_module_t (eight callbacks such as create_loc_conf and merge_loc_conf). The core looks at type and casts ctx to the right struct. C has no inheritance, so this imitates polymorphism with "common header + void * + a type tag" (if C structs and function pointers feel rusty, httpd Part 2 may help).

Module types are four letters

The values that go into type are defined like this.

Split the values into bytes and they are ASCII letters. 0x45524F43 is 45 52 4F 43 = E R O C, and on a CPU that stores the low byte first (little-endian, like x86) it sits in memory as 43 4F 52 45 = C O R E. So a memory dump of a module struct shows the word "CORE" in plain sight. Any number would do to tell types apart, but this choice makes them easy to recognize in a debugger.

A CORE module opens a block; another type handles what's inside

Who handles the http {} block? Not an HTTP module, but the CORE module ngx_http_module.

static ngx_command_t  ngx_http_commands[] = {

    { ngx_string("http"),
      NGX_MAIN_CONF|NGX_CONF_BLOCK|NGX_CONF_NOARGS,
      ngx_http_block,
      0,
      0,
      NULL },

      ngx_null_command
};


static ngx_core_module_t  ngx_http_module_ctx = {
    ngx_string("http"),
    NULL,
    NULL
};


ngx_module_t  ngx_http_module = {
    NGX_MODULE_V1,
    &ngx_http_module_ctx,                  /* module context */
    ngx_http_commands,                     /* module directives */
    NGX_CORE_MODULE,                       /* module type */
    NULL,                                  /* init master */
    NULL,                                  /* init module */
    NULL,                                  /* init process */
    NULL,                                  /* init thread */
    NULL,                                  /* exit thread */
    NULL,                                  /* exit process */
    NULL,                                  /* exit master */
    NGX_MODULE_V1_PADDING
};

src/http/ngx_http.c lines 86–119

This is the typical shape of a module definition: a directive table (an array of ngx_command_t ending with ngx_null_command), a module context, and an ngx_module_t tying them together. The leading NGX_MODULE_V1 macro sets ctx_index and index to "not assigned yet" and name to NULL (src/core/ngx_module.h lines 216–218).

This module has one directive, http: a block (NGX_CONF_BLOCK) at the top level (NGX_MAIN_CONF) taking no arguments (NGX_CONF_NOARGS). When the parser meets http {, it calls ngx_http_block(), which counts the HTTP-type modules (ngx_count_modules(cf->cycle, NGX_HTTP_MODULE)), has them create their config structs, and then parses the block's contents (section 5). events {} is opened the same way by the CORE module ngx_events_module (src/event/ngx_event.c lines 84–89), stream {} by ngx_stream_module (src/stream/ngx_stream.c lines 57–62) and mail {} by ngx_mail_module (src/mail/ngx_mail.c lines 32–37).

Blue boxes are CORE modules that open a block. Directives inside a block are handled only by modules of that block's type. The counts are this build's, from Example 2.

One row of the directive table: ngx_command_t

Each element of the commands array looks like this.

struct ngx_command_s {
    ngx_str_t             name;
    ngx_uint_t            type;
    char               *(*set)(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
    ngx_uint_t            conf;
    ngx_uint_t            offset;
    void                 *post;
};

src/core/ngx_conf_file.h lines 77–84

One line of nginx.conf ultimately means "find the ngx_command_t with that name and call its set function". Many directives use a standard setter for set, such as ngx_conf_set_flag_slot (on/off) or ngx_conf_set_msec_slot (time), and only give the field position through offset. For example, master_process is ngx_conf_set_flag_slot with offsetof(ngx_core_conf_t, master) (src/core/nginx.c lines 51–56). How a directive line becomes a field of a config struct is covered in Part 5.

Example 2: counting this build's modules by type

The module array ngx_modules[] and the name array ngx_module_names[] are in objs/ngx_modules.c, which configure generates (Part 1). I opened the binary in gdb without running nginx and walked the array, printing the four bytes of type as letters, low byte first.

# Example written for this post (not part of the nginx source)
# Walk the module array (generated as objs/ngx_modules.c) in the nginx binary before it runs, printing
# index, name (ngx_module_names[]), the type value in hex and its four bytes as characters.
set $i = 0
while ngx_modules[$i] != 0
  set $t = ngx_modules[$i]->type
  printf "%2d %-36s 0x%08lx %c%c%c%c\n", $i, ngx_module_names[$i], $t, $t & 0xff, ($t >> 8) & 0xff, ($t >> 16) & 0xff, ($t >> 24) & 0xff
  set $i = $i + 1
end

This is ex/u02-architecture/modules.gdb, run with gdb -batch -x /work/modules.gdb /opt/nginx/sbin/nginx.

=== 1. modules compiled into this binary (from ngx_modules[] / ngx_module_names[]) ===
 0 ngx_core_module                      0x45524f43 CORE
 1 ngx_errlog_module                    0x45524f43 CORE
 2 ngx_conf_module                      0x464e4f43 CONF
 3 ngx_openssl_module                   0x45524f43 CORE
 4 ngx_openssl_cache_module             0x45524f43 CORE
 5 ngx_regex_module                     0x45524f43 CORE
 6 ngx_events_module                    0x45524f43 CORE
 7 ngx_event_core_module                0x544e5645 EVNT
 8 ngx_epoll_module                     0x544e5645 EVNT
 9 ngx_thread_pool_module               0x45524f43 CORE
10 ngx_http_module                      0x45524f43 CORE
11 ngx_http_core_module                 0x50545448 HTTP
12 ngx_http_log_module                  0x50545448 HTTP
...
66 ngx_http_range_body_filter_module    0x50545448 HTTP
67 ngx_http_not_modified_filter_module  0x50545448 HTTP
68 ngx_stream_module                    0x45524f43 CORE
69 ngx_stream_core_module               0x4d525453 STRM
70 ngx_stream_log_module                0x4d525453 STRM
...
88 ngx_stream_upstream_zone_module      0x4d525453 STRM
--- count by type ---
     57 HTTP
     20 STRM
      9 CORE
      2 EVNT
      1 CONF
total: 89

• This build has 89 modules: HTTP 57, STREAM 20, CORE 9, EVENT 2, CONF 1. It was not built --with-mail, so there are no MAIL modules.
• The order follows a pattern. Right after each CORE module that opens a block (ngx_events_module at 6, ngx_http_module at 10, ngx_stream_module at 68) comes that block's "core" module (ngx_event_core_module, ngx_http_core_module, ngx_stream_core_module). The order among HTTP modules also determines the order of the filter chain (Part 10).
• The only event method module is ngx_epoll_module, because configure picked what is available on Linux (the select and poll modules are not in this build).

Easy to confuse: name is empty before nginx runs

At first my gdb script tried to print ngx_modules[$i]->name. But that field is NULL before nginx runs. The NGX_MODULE_V1 we saw above leaves name as NULL, and once nginx starts, ngx_preinit_modules() fills in index and name.

    for (i = 0; ngx_modules[i]; i++) {
        ngx_modules[i]->index = i;
        ngx_modules[i]->name = ngx_module_names[i];
    }

src/core/ngx_module.c lines 30–33

So the example reads ngx_module_names[], which is generated in the same order. index (position in the global array) is also set here, while ctx_index (position among modules of the same type) is set by ngx_count_modules() when a block such as http {} is parsed. The next section shows what the two numbers are for.

4. ngx_cycle_t — everything about a running nginx

What hangs off the cycle

struct ngx_cycle_s {
    void                  ****conf_ctx;
    ngx_pool_t               *pool;

    ngx_log_t                *log;
    ngx_log_t                 new_log;
    /* ... */
    ngx_connection_t         *free_connections;
    ngx_uint_t                free_connection_n;

    ngx_module_t            **modules;
    ngx_uint_t                modules_n;
    /* ... */
    ngx_array_t               listening;
    ngx_array_t               paths;
    /* ... */
    ngx_list_t                open_files;
    ngx_list_t                shared_memory;

    ngx_uint_t                connection_n;
    ngx_uint_t                files_n;

    ngx_connection_t         *connections;
    ngx_event_t              *read_events;
    ngx_event_t              *write_events;

    ngx_cycle_t              *old_cycle;

    ngx_str_t                 conf_file;
    /* ... */
};

src/core/ngx_cycle.h lines 39–86

The current cycle is pointed to by the global variable ngx_cycle (in the master code of section 2, ngx_cycle = cycle; after a reload). Most functions take cycle as their first argument because this struct is "the nginx of right now".

conf_ctx: why four levels of pointers

void ****conf_ctx is probably the most famous line in the nginx source. It has four stars because what hangs off each slot has a different shape for each module type. It starts in ngx_init_cycle().

    cycle->conf_ctx = ngx_pcalloc(pool, ngx_max_module * sizeof(void *));

src/core/ngx_cycle.c line 200

The number of slots is ngx_max_module (static modules + 128 slots for dynamic modules, src/core/ngx_module.c line 36), and the slot number is the module's index. Right after that, the CORE modules' create_conf fills in their own slots.

    for (i = 0; cycle->modules[i]; i++) {
        if (cycle->modules[i]->type != NGX_CORE_MODULE) {
            continue;
        }

        module = cycle->modules[i]->ctx;

        if (module->create_conf) {
            rv = module->create_conf(cycle);
            if (rv == NULL) {
                ngx_destroy_pool(pool);
                return NULL;
            }
            cycle->conf_ctx[cycle->modules[i]->index] = rv;
        }
    }

src/core/ngx_cycle.c lines 233–248

ngx_core_module's slot (0) gets an ngx_core_conf_t (worker_processes, user, pid …) directly. But ngx_http_module has a NULL create_conf (see ngx_http_module_ctx in the section 3 excerpt), so its slot stays empty for now. It is filled by ngx_http_block() when the parser meets http {. What goes in is not a single config struct but a struct pointing to three arrays.

typedef struct {
    void        **main_conf;
    void        **srv_conf;
    void        **loc_conf;
} ngx_http_conf_ctx_t;

src/http/ngx_http_config.h lines 17–21

Each of the three arrays has as many slots as there are HTTP modules (ngx_http_max_module), and the slot number is the module's ctx_index (src/http/ngx_http.c lines 150–181). For each server {}, main_conf points to the http one as is and only new srv_conf and loc_conf arrays are created (src/http/ngx_http_core_module.c lines 3037–3048), and each location {} likewise gets its own loc_conf array. The events {} slot is the in-between shape: one void * slot pointing to the event modules' array (src/event/ngx_event.c lines 973–983).

The point is the two levels of slot numbers: conf_ctx by index, the per-type arrays below it by ctx_index. 0, 10 and 57 are this build's values, printed in Example 3.

That is why an HTTP module handling a request fetches its config like this:

#define ngx_http_get_module_main_conf(r, module)                             \
    (r)->main_conf[module.ctx_index]
#define ngx_http_get_module_srv_conf(r, module)  (r)->srv_conf[module.ctx_index]
#define ngx_http_get_module_loc_conf(r, module)  (r)->loc_conf[module.ctx_index]

src/http/ngx_http_config.h lines 55–58

The request struct r holds "the arrays of the server and location this request matched", and each module looks only at its own ctx_index slot. The result of finding a location is that r->loc_conf points to a different array depending on which location the request matched (Part 9). It's the same idea as httpd's ap_get_module_config(r->per_dir_config, &my_module) finding a void * array slot by module number (httpd Part 3).

Example 3: looking at the cycle from inside a worker

I attached gdb to a running worker and printed cycle fields and module numbers.

gdb -batch -p "$WORKER" \
    -ex 'p ngx_cycle->modules_n' \
    -ex 'p ngx_cycle->connection_n' \
    -ex 'p ngx_cycle->free_connection_n' \
    -ex 'p ngx_cycle->listening.nelts' \
    -ex 'p ngx_core_module.index' \
    -ex 'p ((ngx_core_conf_t *) ngx_cycle->conf_ctx[ngx_core_module.index])->worker_processes' \
    -ex 'p ngx_http_module.index' \
    -ex 'p ngx_http_core_module.index' \
    -ex 'p ngx_http_core_module.ctx_index' \
    -ex 'p ngx_http_max_module' \
    -ex 'p ngx_event_max_module'

=== 6. the cycle seen from a worker (gdb print) ===
$1 = 89
$2 = 1024
$3 = 1022
$4 = 1
$5 = 0
$6 = 2
$7 = 10
$8 = 11
$9 = 0
$10 = 57
$11 = 2

I checked in the source why free_connection_n is 1022. When a worker starts, ngx_event_process_init() takes one connection per listening socket with ngx_get_connection() (src/event/ngx_event.c line 809), and ngx_add_channel_event() takes one for the channel socket (src/os/unix/ngx_channel.c line 215). In nginx a "connection" (ngx_connection_t) means not just a client connection but any socket the event loop watches. So worker_connections also covers listening sockets and, when proxying, backend connections.

Easy to confuse: the default for worker_connections is 512

Distribution nginx.conf files often say worker_connections 1024;, but the default when you leave it out is 512. ngx_event_core_init_conf() sets it with ngx_conf_init_uint_value(ecf->connections, DEFAULT_CONNECTIONS); (src/event/ngx_event.c lines 1330–1331), and DEFAULT_CONNECTIONS is 512 (line 13). The same function sets accept_mutex to off by default (ngx_conf_init_value(ecf->accept_mutex, 0);, line 1339). If you remember reading "accept_mutex is on by default" in an older article, that is not what this version's source says. How this affects connection distribution is in Part 7.

5. Config contexts — main, events, http, server, location

A context is a pair of values

The config parser knows which block it is reading through two fields of ngx_conf_t:

• module_type — the module type whose directives this block accepts (CORE, EVNT, HTTP …)
• cmd_type — this block's context bit (NGX_MAIN_CONF, NGX_HTTP_SRV_CONF …)

The functions that open blocks change both right before parsing the block's contents.

Context bits live in the top byte of the type field. The comment at the top of ngx_conf_file.h describes type as "TT FF AAAA" (context · flags · argument count) (src/core/ngx_conf_file.h lines 16–20).

#define NGX_HTTP_MAIN_CONF        0x02000000
#define NGX_HTTP_SRV_CONF         0x04000000
#define NGX_HTTP_LOC_CONF         0x08000000
#define NGX_HTTP_UPS_CONF         0x10000000
#define NGX_HTTP_SIF_CONF         0x20000000
#define NGX_HTTP_LIF_CONF         0x40000000
#define NGX_HTTP_LMT_CONF         0x80000000

src/http/ngx_http_config.h lines 41–47

SIF is if inside server, LIF is if inside location, and LMT is the limit_except block. A directive ORs together the bits of every context it may appear in. A few examples show that the "Context:" line in the nginx docs is exactly these bits.

Note that events and http use the same context bit, 0x02000000. The bit alone cannot tell them apart, so the module type is checked as well.

Where the check happens: ngx_conf_handler()

ngx_conf_handler() is called for every line of tokens read (a directive name and its arguments). The exception is the inside of blocks such as types {} and map {}, whose opening function installs its own handler (cf->handler); lines in there go to that function and skip this check (src/core/ngx_conf_file.c lines 298–325).

    for (i = 0; cf->cycle->modules[i]; i++) {

        cmd = cf->cycle->modules[i]->commands;
        if (cmd == NULL) {
            continue;
        }

        for ( /* void */ ; cmd->name.len; cmd++) {

            if (name->len != cmd->name.len) {
                continue;
            }

            if (ngx_strcmp(name->data, cmd->name.data) != 0) {
                continue;
            }

            found = 1;

            if (cf->cycle->modules[i]->type != NGX_CONF_MODULE
                && cf->cycle->modules[i]->type != cf->module_type)
            {
                continue;
            }

            /* is the directive's location right ? */

            if (!(cmd->type & cf->cmd_type)) {
                continue;
            }
            /* ... */
            /* set up the directive's configuration context */

            conf = NULL;

            if (cmd->type & NGX_DIRECT_CONF) {
                conf = ((void **) cf->ctx)[cf->cycle->modules[i]->index];

            } else if (cmd->type & NGX_MAIN_CONF) {
                conf = &(((void **) cf->ctx)[cf->cycle->modules[i]->index]);

            } else if (cf->ctx) {
                confp = *(void **) ((char *) cf->ctx + cmd->conf);

                if (confp) {
                    conf = confp[cf->cycle->modules[i]->ctx_index];
                }
            }

            rv = cmd->set(cf, cmd, conf);
            /* ... */
        }
    }

    if (found) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "\"%s\" directive is not allowed here", name->data);

        return NGX_ERROR;
    }

    ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                       "unknown directive \"%s\"", name->data);

src/core/ngx_conf_file.c lines 374–494

1. It scans every module's directive table from the start looking for the name. It's a linear search, not a hash — this happens once while reading the config, so speed doesn't matter.
2. On a name match it sets found = 1 and checks two things: is the module's type the current block's module_type (CONF modules are exempt — which is why include works anywhere), and does the directive's type contain the current block's cmd_type bit?
3. If both pass, it finds the config struct and passes it to set. This is where the two numbers from section 4 come in: top-level directives use conf_ctx[index]; directives inside an HTTP block pick an array with cmd->conf (the offsetof of main, srv or loc) and then use slot [ctx_index].
4. If nothing matches, it reports "not allowed here" or "unknown directive" depending on whether the name existed at all (found).

Example 4: directives in the wrong context, and nginx -T

I made a small config with just events { } and server { listen 8081; }, put four directives one at a time directly inside http {} (line 5 of the file), and ran nginx -t (an option that only checks the config and exits).

=== 2. directives in the wrong context (nginx -t) ===
--- core directive: http { worker_processes 4; } ---
nginx: [emerg] "worker_processes" directive is not allowed here in /tmp/u02/bad.conf:5
--- event directive: http { worker_connections 512; } ---
nginx: [emerg] "worker_connections" directive is not allowed here in /tmp/u02/bad.conf:5
--- loc directive: http { proxy_pass http://127.0.0.1:9000; } ---
nginx: [emerg] "proxy_pass" directive is not allowed here in /tmp/u02/bad.conf:5
--- typo: http { proxy_pas http://127.0.0.1:9000; } ---
nginx: [emerg] unknown directive "proxy_pas" in /tmp/u02/bad.conf:5

• worker_processes is a CORE module's directive, so in an HTTP block it fails the module type check.
• worker_connections is the most interesting. Its context bit NGX_EVENT_CONF (0x02000000) is the same value as http {}'s cmd_type, NGX_HTTP_MAIN_CONF (0x02000000), so a bit check alone would have let it through. It fails because the module type differs (EVNT ≠ HTTP). The design reuses the same bits for each type and tells them apart with the type check.
• proxy_pass is an HTTP module, so it passes the type check, but its type lacks the NGX_HTTP_MAIN_CONF bit, so it fails the context check.
• The typo matches no directive name at all (found == 0), hence "unknown directive". The difference between the two messages is exactly the found flag.

nginx -T checks the config and also prints every config file it read (including included ones) verbatim under # configuration file …: headers. It's the option you use to see "the config actually loaded" on a running server.

=== 4. nginx -T (first lines) ===
nginx: the configuration file /work/nginx.conf syntax is ok
nginx: configuration file /work/nginx.conf test is successful
# configuration file /work/nginx.conf:
# 이 글을 위해 쓴 예제입니다 (nginx 의 소스가 아닙니다)
# 2편: 마스터 하나 + 워커 둘, events·http·server·location 문맥을 하나씩 둔 가장 작은 설정
worker_processes 2;
error_log /tmp/u02/error.log notice;
pid /tmp/u02/nginx.pid;

(The two Korean lines in the output are the comment header of the actual config file used for the run; they say the same as the comment in the config shown in Example 1.) The "syntax is ok" line is printed by ngx_init_cycle() from section 4 right after ngx_conf_parse() finishes (src/core/ngx_cycle.c lines 292–295). The mechanism that keeps file contents for dumping (cycle->config_dump) and the order in which -t and -T are handled are in Part 4.

Easy to confuse: values set in http are not "inherited" by server and location at request time

Set keepalive_timeout once in http {} and it applies to every location. It's tempting to think "the inner block looks up the outer block's value", but the source doesn't do that. As section 4 showed, it builds a separate config struct for each http {}, server {} and location {}, and after the whole file is read, the merge_srv_conf and merge_loc_conf callbacks fill in "if the inner one is unset, copy the outer one" ahead of time. At request time nothing looks upward; each request looks only at its own loc_conf. How this merge runs is in Part 5.

6. The event loop inside one worker

The order of one turn

This is ngx_process_events_and_timers(), which the worker loop in section 2 kept calling.

void
ngx_process_events_and_timers(ngx_cycle_t *cycle)
{
    ngx_uint_t  flags;
    ngx_msec_t  timer, delta;

    if (ngx_timer_resolution) {
        timer = NGX_TIMER_INFINITE;
        flags = 0;

    } else {
        timer = ngx_event_find_timer();
        flags = NGX_UPDATE_TIME;
        /* ... */
    }

    if (ngx_use_accept_mutex) {
        /* ... */
    }

    if (!ngx_queue_empty(&ngx_posted_next_events)) {
        ngx_event_move_posted_next(cycle);
        timer = 0;
    }

    delta = ngx_current_msec;

    (void) ngx_process_events(cycle, timer, flags);

    delta = ngx_current_msec - delta;

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                   "timer delta: %M", delta);

    ngx_event_process_posted(cycle, &ngx_posted_accept_events);

    if (ngx_accept_mutex_held) {
        ngx_shmtx_unlock(&ngx_accept_mutex);
    }

    ngx_event_expire_timers();

    ngx_event_process_posted(cycle, &ngx_posted_events);
}

src/event/ngx_event.c lines 194–264

This is all a worker does. Step 2 is the only place it sleeps; everything else must finish without blocking.

1. How long to sleep — ngx_event_find_timer() returns the milliseconds until the nearest timer (timeout). Timers live in a red-black tree (a balanced binary tree where insert, delete and find-minimum are all fast). The timer=18446744073709551615 in Example 1 is the value when no timer is pending.
2. Wait — ngx_process_events is not a function but a macro for the function pointer ngx_event_actions.process_events (src/event/ngx_event.h line 371). The epoll module plugs in its own functions at init with ngx_event_actions = ngx_epoll_module_ctx.actions; (src/event/modules/ngx_epoll_module.c line 369), so on Linux this becomes ngx_epoll_process_events() → epoll_wait(). On FreeBSD the kqueue module plugs into the same spot.
3. Handle — for every event epoll returns, it calls that event's handler function pointer (rev->handler(rev);, src/event/modules/ngx_epoll_module.c line 901). For a listening socket that's ngx_event_accept; for an HTTP connection it's the function that reads the request (section 7). Only when holding accept_mutex does it put events into a posted queue (a list of events to handle later) instead of calling them right away, and handle them in steps 4 and 6.
4. –6. Cleanup — run the queued accept events, the timers that ran out (this is where things like proxy_read_timeout fire), and the queued ordinary events, in that order.

The whole design rests on the promise that a handler never holds on for long. If there's no data to read yet, a non-blocking recv() returns EAGAIN, and the handler registers "call me again when this becomes readable" with epoll and returns immediately. That is how one thread can take turns serving thousands of connections. Conversely, if one handler reads the disk for a long time or does heavy computation, every other connection in that worker stalls — which is why nginx has aio and thread pools (ngx_thread_pool_module, number 9 in Example 2). We read one turn line by line in Part 7.

Example 5: 300 connections, one thread

I checked whether one thread really holds hundreds of connections. A Python script opened 300 TCP connections to port 8080 and held them without sending any request; then I printed stub_status, each worker's thread count, and, via gdb, the number of connections each worker is using (connection_n - free_connection_n).

# Example written for this post (not part of the nginx source)
# Open N TCP connections to 8080 and hold them (no request is sent); exit when stdin is closed.
import socket, sys

n = int(sys.argv[1])
socks = []
for _ in range(n):
    s = socket.create_connection(("127.0.0.1", 8080))
    socks.append(s)
print("opened %d idle connections" % len(socks), flush=True)
sys.stdin.read()
for s in socks:
    s.close()

=== 7. 300 idle connections, still one thread per worker ===
opened 300 idle connections
Active connections: 301 
server accepts handled requests
 303 303 3 
Reading: 0 Writing: 1 Waiting: 300 
worker 46: threads=1 connections in use=126
worker 47: threads=1 connections in use=178

• Active connections: 301 is the 300 held connections plus the curl connection asking stub_status; that one is Writing: 1 (writing its response) and the 300 are Waiting: 300.
• Both workers have one thread. It doesn't grow with the number of connections.
• The workers use 126 + 178 = 304 connections. Subtracting the baseline 2 each (listening socket and channel) from Example 3 gives 124 + 176 = 300, matching the held connections.
• The split between the two workers was uneven and changed from run to run (an earlier run gave 51 : 249). Which worker accepts a connection is covered with accept_mutex and reuseport in Part 7.

7. HTTP, stream and mail — three layers on the same core

Finally, let's stack the folders into layers.

Upper layers use lower ones. The three protocols meet the event core at a single function pointer: the listening socket's handler.

The event core knows nothing about HTTP. ngx_event_accept(), which accepts connections, creates the new connection and then just calls the function hung on the listening socket.

        ls->handler(c);

src/event/ngx_event_accept.c line 312

Each protocol fills in that function pointer. Sockets created by listen inside http {} get ls->handler = ngx_http_init_connection; (src/http/ngx_http.c line 1882), those inside stream {} get ngx_stream_init_connection (src/stream/ngx_stream.c line 1002), and mail {} gets ngx_mail_init_connection (src/mail/ngx_mail.c line 326). This one line is why, on Part 1's "main road of a request", ngx_http_init_connection came right after ngx_event_accept.

stream carries HTTP's structure over to TCP/UDP almost unchanged. Its context bits also restart at 0x02000000 just like HTTP's (NGX_STREAM_MAIN_CONF and friends, src/stream/ngx_stream.h lines 317–319) — section 5's "same bits, told apart by the type check" applies here too. This series follows HTTP, and Part 12 places stream side by side with it.

Easy to confuse: looking for httpd's MPM, you won't find one

If you've read httpd, you'll go looking for "something like an MPM that swaps how connections are accepted". nginx has no such module. The process structure (one master + N workers, one thread each) is fixed in src/os/unix/ngx_process_cycle.c; the only swappable thing is the event method below it (ngx_epoll_module, kqueue and so on — the use directive in events {}). Where httpd's event MPM arrived at "hand idle keep-alive connections to epoll and attach a thread only for work", nginx started from "hand every connection to epoll, with one thread".

What this post leaves out

To keep the map to one page, I left out:

• The front part of main() — command-line options (ngx_get_options()), inherited sockets (ngx_add_inherited_sockets()), daemonizing (ngx_daemon()). src/core/nginx.c — Part 4
• The rest of ngx_init_cycle() — shared memory, opening listening sockets, init_module. src/core/ngx_cycle.c — Part 4
• The cache manager and cache loader processes (ngx_start_cache_manager_processes(), ngx_cache_manager_process_cycle()). src/os/unix/ngx_process_cycle.c line 375, and from line 1139
• Binary upgrade with USR2 (ngx_exec_new_binary()) and ngx_reap_children(), which restarts dead workers — Part 6
• Dynamic modules (load_module, ngx_add_module()) and the module signature (NGX_MODULE_SIGNATURE). src/core/ngx_module.[ch] — Part 5
• The eight callbacks of ngx_http_module_t and merging. src/http/ngx_http_config.h lines 24–36 — Part 5
• ngx_conf_read_token(), which splits config tokens. src/core/ngx_conf_file.c from line 509 — Part 4
• QUIC and HTTP/3 (src/event/quic, src/http/v3) and the Windows port (src/os/win32). Not enabled in this series' build.
• The #if (NGX_CONTROL_API) block in the master loop (ngx_process_cycle.c lines 177–197). The feature isn't used in this build, so I did not look into it.

Summary

What's next

We fold the map and start with what every box on it sits on. The next post is about memory pools and core data structures. In this post we saw the cycle, config structs and connections all created with ngx_pcalloc(pool, …). Next we check with debug logs and gdb how that pool splits small and large allocations, how a request's memory disappears all at once when it ends, and what the ngx_str_t, ngx_array_t, ngx_buf_t and ngx_chain_t we passed several times here look like.

Digging into nginx's Source 3: Memory Pools and Core Data Structures — ngx_pool_t, ngx_str_t, ngx_buf_t and ngx_chain_t

The two earlier series unfolded a big-picture map at the same point. In httpd Part 3, httpd's core defined hooks and the MPM and modules hung functions on them, and each module kept config create/merge functions and a directive table in one module struct. nginx's ngx_module_t and ngx_command_t look very much like that, but the part that accepts connections is not a swappable MPM — it's a fixed master/worker setup plus an event loop (httpd Part 8). On the Tomcat Part 2 map, server.xml elements became a parent–child tree of Java objects. nginx.conf blocks nest too, but they become pointer arrays rooted at conf_ctx instead of an object tree. And unlike Tomcat's NIO endpoint, which watches many connections with one selector and hands the work to a thread pool, an nginx worker watches and works on the same thread.