Digging into nginx's Source 7: The Event Loop — epoll, Timers, Posted Events and accept

What this post covers

This is the seventh post in the "Digging into nginx's Source" series. In Part 6 the master started the workers and each worker entered the endless loop of ngx_worker_process_cycle(). The body of that loop is really one line: ngx_process_events_and_timers(cycle);. This post goes inside that line.

On the map from Part 2, this post zooms into the Event loop box in area B. Processes and events.

The green box is this post. Part 6 started the workers; from Part 8 on we look at the HTTP handlers this loop calls.

Four questions drive the post:

• How does a worker serve thousands of connections in turn with a single thread — what exactly happens, in what order, in one turn of the loop?
• What does events { use epoll; worker_connections 1024; } turn into in the source — how is epoll plugged in, and what is worker_connections actually counting?
• Who measures timeouts like client_header_timeout and keepalive_timeout, and how do they mesh with epoll_wait?
• With several workers, who accepts a new connection — what do accept_mutex and reuseport change?

All line numbers in this post refer to this commit. Part 1 shows how to check out the same commit and build the Docker image used for the examples.

The src/event folder also holds the other event-method modules (epoll, kqueue, poll, select …), the OpenSSL glue, UDP, proxy buffering (event pipe) and QUIC. This post picks only the main path on Linux epoll: a TCP connection being accepted, waited on, and closed when its time runs out. What I left out is listed by name and location in "What this post leaves out" at the end.

Terms to know first

An event loop is a loop that asks the operating system "has anything become ready?", handles only what has, and asks again. An nginx worker lives inside one such loop. Part 2 showed its shape in one picture; this post opens each step of one turn.

A non-blocking socket is one where recv() does not wait when there is no data but returns EAGAIN ("nothing right now") immediately. To run many connections on one thread you can never stall on any single connection, so every nginx connection socket is non-blocking.

epoll is the Linux facility that "watches thousands of sockets at once and tells you only about the ready ones". epoll_create makes a watch list, epoll_ctl adds and removes sockets, and epoll_wait sleeps until something is ready (or a given time passes). There are two notification styles. Level-triggered keeps telling you as long as data remains; edge-triggered (EPOLLET) tells you once, when the state changes from not readable to readable.

An event (ngx_event_t) is a struct holding "the function to call when this socket has something to read (or room to write)". Its key fields are handler (the function pointer to call) and data (usually the connection). Each connection (ngx_connection_t) carries a read event c->read and a write event c->write, and the same struct doubles as a timer.

A timer is a request: "call this event as timed out in N milliseconds". All of a worker's timers live in one red-black tree (a balanced binary tree where insert, delete and find-minimum are all fast). The tree itself was covered in Part 3.

A posted event is one put on a queue meaning "don't call it right now, call it before this turn ends" (section 6).

Connection slot: in this post, one element of the cycle->connections array. nginx does not malloc connection structs on demand; each worker builds worker_connections of them up front and lends them out (section 3).

1. One line in the worker's for loop — ngx_process_events_and_timers()

The worker's for ( ;; ) from Part 6 puts the single line ngx_process_events_and_timers(cycle); between checks of the exit flags (ngx_exiting, ngx_terminate, ngx_quit …) (src/os/unix/ngx_process_cycle.c lines 740–778). ngx_single_process_cycle() for master_process off; (line 322) and the cache manager process (line 1184) call the same function. The event loop is the engine every working nginx process runs on.

Eight steps in one turn

Here is the whole function. It is about 70 lines.

ngx_process_events_and_timers(ngx_cycle_t *cycle)
{
    ngx_uint_t  flags;
    ngx_msec_t  timer, delta;

    if (ngx_timer_resolution) {
        timer = NGX_TIMER_INFINITE;
        flags = 0;

    } else {
        timer = ngx_event_find_timer();
        flags = NGX_UPDATE_TIME;

        /* ... Windows-only block (#if (NGX_WIN32)) ... */
    }

    if (ngx_use_accept_mutex) {
        if (ngx_accept_disabled > 0) {
            ngx_accept_disabled--;

        } else {
            if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) {
                return;
            }

            if (ngx_accept_mutex_held) {
                flags |= NGX_POST_EVENTS;

            } else {
                if (timer == NGX_TIMER_INFINITE
| timer > ngx_accept_mutex_delay)
                {
                    timer = ngx_accept_mutex_delay;
                }
            }
        }
    }

    if (!ngx_queue_empty(&ngx_posted_next_events)) {
        ngx_event_move_posted_next(cycle);
        timer = 0;
    }

    delta = ngx_current_msec;

    (void) ngx_process_events(cycle, timer, flags);

    delta = ngx_current_msec - delta;

    ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                   "timer delta: %M", delta);

    ngx_event_process_posted(cycle, &ngx_posted_accept_events);

    if (ngx_accept_mutex_held) {
        ngx_shmtx_unlock(&ngx_accept_mutex);
    }

    ngx_event_expire_timers();

    ngx_event_process_posted(cycle, &ngx_posted_events);
}

src/event/ngx_event.c lines 195–264

Yellow boxes touch the timer tree; grey ones are skipped depending on configuration. The only place the loop waits (sleeps) is the green box.

The ordering carries these ideas:

1. Decide how long to sleep first. If the nearest timer is 3 seconds away, epoll_wait must sleep at most 3 seconds so that, even if nothing happens, it wakes up in time to handle that timer. With no timers at all it sleeps indefinitely (section 5).
2. The accept mutex is off by default, so this is usually skipped (section 7).
3. The posted next queue holds "handle me next turn" items. If there are any, timer = 0 makes epoll_wait return at once instead of sleeping (section 6).
4. Wait. ngx_process_events is not a function but a function-pointer macro (section 2). With epoll it calls epoll_wait, and on wake-up calls the handler of every ready event (section 4).
5. – 8. Clean-up. Posted accept queue → release the mutex → expired timers → posted queue. Why accepts come first and the mutex is released right after is in section 7; why timers come after epoll is in section 5.

Handlers make one promise: never wait. With nothing to read they get EAGAIN, hand the socket back to epoll with "call me again when it's readable", and return immediately. If one handler hangs on to the thread for disk I/O or heavy computation, every other connection in the same worker stalls. That is the price of not splitting work across threads.

Easy to get wrong: delta is only used for logging now

delta is the time difference around epoll_wait, and in this version it only feeds the debug line timer delta: …. Older versions checked timers only when the clock had moved, as in if (delta) { ngx_event_expire_timers(); }. When an already-expired timer was left in the tree, epoll_wait(…, 0) returned immediately with the clock unchanged, so the timer was never processed and the loop spun; commit c108f04e8 (2021-03-26, "Events: fixed expiration of timers in the past.", from 1.19.9) removed the condition. If you see if (delta) in an article about an old version, it no longer matches this code.

timer_resolution changes step 1. Instead of looking at the timers and sleeping indefinitely, nginx uses setitimer() to get a SIGALRM at that interval, which wakes epoll_wait (src/event/ngx_event.c lines 696–719). It is off by default.

2. Where epoll plugs in — the ngx_event_actions function-pointer table

The string epoll_wait does not appear in ngx_event.c. The event core calls "add, delete, wait" only through a table of function pointers, and the table is filled by an OS-specific event module (src/event/modules/). It is the same role APR's pollset played in hiding OS differences in httpd Part 8.

typedef struct {
    ngx_int_t  (*add)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);
    ngx_int_t  (*del)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);

    ngx_int_t  (*enable)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);
    ngx_int_t  (*disable)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);

    ngx_int_t  (*add_conn)(ngx_connection_t *c);
    ngx_int_t  (*del_conn)(ngx_connection_t *c, ngx_uint_t flags);

    ngx_int_t  (*notify)(ngx_event_handler_pt handler);

    ngx_int_t  (*process_events)(ngx_cycle_t *cycle, ngx_msec_t timer,
                                 ngx_uint_t flags);

    ngx_int_t  (*init)(ngx_cycle_t *cycle, ngx_msec_t timer);
    void       (*done)(ngx_cycle_t *cycle);
} ngx_event_actions_t;

src/event/ngx_event.h lines 149–166

An event module's context, ngx_event_module_t (the per-module-type struct that ngx_module_t's ctx points to, see Part 5), holds name, create_conf, init_conf and this table as actions (src/event/ngx_event.h lines 417–424). Here is what the epoll module puts in it:

static ngx_event_module_t  ngx_epoll_module_ctx = {
    &epoll_name,
    ngx_epoll_create_conf,               /* create configuration */
    ngx_epoll_init_conf,                 /* init configuration */

    {
        ngx_epoll_add_event,             /* add an event */
        ngx_epoll_del_event,             /* delete an event */
        ngx_epoll_add_event,             /* enable an event */
        ngx_epoll_del_event,             /* disable an event */
        ngx_epoll_add_connection,        /* add an connection */
        ngx_epoll_del_connection,        /* delete an connection */
#if (NGX_HAVE_EVENTFD)
        ngx_epoll_notify,                /* trigger a notify */
#else
        NULL,                            /* trigger a notify */
#endif
        ngx_epoll_process_events,        /* process the events */
        ngx_epoll_init,                  /* init the events */
        ngx_epoll_done,                  /* done the events */
    }
};

src/event/modules/ngx_epoll_module.c lines 179–200

The enable slot holds the same function as add, and disable the same as del. Unlike kqueue, which supports "temporarily disable" separately, for epoll that is the same thing as adding and removing.

The top two steps (reading the config) happen in the master; everything below happens in each worker. The yellow box is a global variable, one per worker.

Choosing happens when the config is read. If events {} has no use, ngx_event_core_init_conf() actually calls epoll_create(100) once and picks the epoll module if that works (kqueue on systems that have it, select if there is nothing else — src/event/ngx_event.c lines 1272–1304). The chosen module's ctx_index becomes ecf->use.

Plugging in happens as each worker starts. ngx_event_process_init(), the event core module's init_process callback, finds the event module with ctx_index == ecf->use in the module list and calls module->actions.init(cycle, ngx_timer_resolution) (lines 675–692). ngx_epoll_init() creates the epoll instance with epoll_create(cycle->connection_n / 2) (line 330), allocates the result array event_list with epoll_events entries (default 512), then installs the table and sets flags describing "what kind of method this is":

    ngx_event_actions = ngx_epoll_module_ctx.actions;

#if (NGX_HAVE_CLEAR_EVENT)
    ngx_event_flags = NGX_USE_CLEAR_EVENT
#else
    ngx_event_flags = NGX_USE_LEVEL_EVENT
#endif
NGX_USE_GREEDY_EVENT
NGX_USE_EPOLL_EVENT;

src/event/modules/ngx_epoll_module.c lines 369–377

From then on, the core and the HTTP modules call the table through macros:

#define ngx_process_events   ngx_event_actions.process_events
#define ngx_done_events      ngx_event_actions.done

#define ngx_add_event        ngx_event_actions.add
#define ngx_del_event        ngx_event_actions.del
#define ngx_add_conn         ngx_event_actions.add_conn
#define ngx_del_conn         ngx_event_actions.del_conn

#define ngx_notify           ngx_event_actions.notify

src/event/ngx_event.h lines 371–379

So ngx_process_events(cycle, timer, flags) in section 1 is ngx_epoll_process_events(cycle, timer, flags) in this build. The core consults ngx_event_flags when it needs to decide "how should I do this with this method?". For example, with NGX_USE_CLEAR_EVENT (edge-triggered), ngx_handle_read_event() in section 4 leaves an already registered event alone.

Where the events {} directives live in the source

I checked every default against the ngx_conf_init_* lines at the end of ngx_event_core_init_conf() (src/event/ngx_event.c lines 1330–1340).

Part 2 already noted that, unlike the worker_connections 1024; you often see in distribution nginx.conf files, the value when you write nothing is 512. accept_mutex became off by default in commit 46dd747c9 (2016-07-15, from 1.11.3).

Easy to get wrong: each worker installs the table itself

ngx_event_actions is a global variable, but the master does not fill it and pass it down through fork(). The master never calls init_process, so its table stays empty and it has no epoll instance. Each worker having its own epoll instance and its own connection array is where the next section starts (step 6 of the worker initialization order in Part 6).

3. Connection slots — worker_connections is the size of a pre-built array

Two structs

Keeping only the fields the event loop uses, the connection struct looks like this:

struct ngx_connection_s {
    void               *data;
    ngx_event_t        *read;
    ngx_event_t        *write;

    ngx_socket_t        fd;

    ngx_recv_pt         recv;
    ngx_send_pt         send;
    /* ... */

    ngx_pool_t         *pool;

    /* ... */

    ngx_queue_t         queue;

    ngx_atomic_uint_t   number;

    /* ... */

    unsigned            reusable:1;
    unsigned            close:1;
    /* ... */
};

src/core/ngx_connection.h lines 125–206

• data belongs to whoever uses the connection. In HTTP it first points to an ngx_http_connection_t, then to the request once one exists (Part 8). In a free slot it points to the next free slot (below).
• recv/send are OS-specific I/O function pointers. For TLS connections they are swapped for functions that go through OpenSSL.
• number is the connection number you see as *2 in the debug log; queue, reusable and close are used for the list of "connections we may close if we must" (below).

The event struct ngx_event_s starts with a dense run of 1-bit flags, followed by handler, the timer node timer (ngx_rbtree_node_t) and the queue node queue (ngx_queue_t) (src/event/ngx_event.h lines 19–121). The bits that come up often in this post:

And the key point of nginx's event design is that one such struct can hang in three places at once.

Because the timer node and the queue node are embedded in the event struct, setting a timer or posting an event allocates no memory.

To get from a node taken out of the tree or a queue back to the event, nginx uses macros that subtract the offsetof: ngx_rbtree_data(node, ngx_event_t, timer) and ngx_queue_data(q, ngx_event_t, queue) (src/core/ngx_rbtree.h lines 50–51, src/core/ngx_queue.h lines 106–107). This is the "embed the node inside the struct" technique from Part 3.

Built all at once when the worker starts

After installing epoll, ngx_event_process_init() builds three arrays — connections, read events and write events — of connection_n (= worker_connections) elements each, and links up the free list:

    cycle->connections =
        ngx_alloc(sizeof(ngx_connection_t) * cycle->connection_n, cycle->log);
    /* ... read_events and write_events are ngx_alloc'ed with the same size ... */

    rev = cycle->read_events;
    for (i = 0; i < cycle->connection_n; i++) {
        rev[i].closed = 1;
        rev[i].instance = 1;
    }

    /* ... */

    i = cycle->connection_n;
    next = NULL;

    do {
        i--;

        c[i].data = next;
        c[i].read = &cycle->read_events[i];
        c[i].write = &cycle->write_events[i];
        c[i].fd = (ngx_socket_t) -1;

        next = &c[i];
    } while (i);

    cycle->free_connections = next;
    cycle->free_connection_n = cycle->connection_n;

src/event/ngx_event.c lines 750–796

Because it walks backwards linking with c[i].data = next, at the end free_connections points at c[0] and the free list (a singly linked list) runs c[0] → c[1] → … → c[n-1] → NULL. The thrifty part is that there is no separate next pointer: the link is stored in the data field, which a free slot doesn't otherwise use.

Since lending and returning both touch only the head, a slot just returned is the first to be reused. That is why the instance bit in section 4 is needed.

Here is the middle of the lending function, ngx_get_connection():

    c = ngx_cycle->free_connections;

    if (c == NULL) {
        ngx_log_error(NGX_LOG_ALERT, log, 0,
                      "%ui worker_connections are not enough",
                      ngx_cycle->connection_n);

        return NULL;
    }

    ngx_cycle->free_connections = c->data;
    ngx_cycle->free_connection_n--;

    /* ... zero the connection struct and refill read/write/fd/log ... */

    instance = rev->instance;

    ngx_memzero(rev, sizeof(ngx_event_t));
    ngx_memzero(wev, sizeof(ngx_event_t));

    rev->instance = !instance;
    wev->instance = !instance;

src/core/ngx_connection.c lines 1225–1258

While wiping the events to zero, it keeps the instance bit — flipped. Section 4 explains why. The heart of the returning function ngx_free_connection() is three lines: c->data = free_connections; free_connections = c; free_connection_n++;. After them comes a line that clears cycle->files, a table allocated only by fd-indexed methods such as poll and devpoll (lines 1273–1282).

In this build, six places borrow slots: the listening socket (ngx_event_process_init), the channel to the master (ngx_add_channel_event, Part 6), new clients (ngx_event_accept), outgoing upstream connections (ngx_event_connect_peer, Part 11), UDP (ngx_event_recvmsg), and, when a resolver is configured, connections to the DNS server (src/core/ngx_resolver.c) (found by grepping for ngx_get_connection — QUIC has some too, but it is not enabled in this build). So worker_connections is not "the number of clients" but all the connection slots one worker can hold at once. A reverse proxy uses two slots per request (client + upstream).

When slots run short: idle connections go first

ngx_get_connection() calls ngx_drain_connections() first (line 1223):

    if (cycle->free_connection_n > cycle->connection_n / 16
| cycle->reusable_connections_n == 0)
    {
        return;
    }

    /* ... at most once a second, warn "... worker_connections are not enough, reusing connections" ... */

    c = NULL;
    n = ngx_max(ngx_min(32, cycle->reusable_connections_n / 8), 1);

    for (i = 0; i < n; i++) {
        /* ... */

        q = ngx_queue_last(&cycle->reusable_connections_queue);
        c = ngx_queue_data(q, ngx_connection_t, queue);

        /* ... */

        c->close = 1;
        c->read->handler(c->read);
    }

src/core/ngx_connection.c lines 1411–1442

When free slots drop to 1/16 of the total or fewer, it picks a few connections marked with ngx_reusable_connection(c, 1) — ones that haven't sent a request yet, or keep-alive connections waiting for the next one — sets c->close = 1 and calls their read handler. The handler sees close and closes the connection, freeing a slot. New connections go to the front of that list (lines 1393–1394) and this code takes from the back, so the connection idle the longest is closed first.

Example 1: running out of slots with worker_connections 6

This example ran in a Docker container (Debian bookworm, --with-debug) on nginx 1.31.9 built unchanged from the reference commit (9a7439fd79). The example folder is ex/u07-connections.

# An example written for this post (not part of nginx's source)
# worker_connections is set very small (6) to see what happens when slots run out
worker_processes 1;
error_log /tmp/u07/error.log warn;
pid /tmp/u07/nginx.pid;

events {
    worker_connections 6;
}

http {
    access_log off;

    server {
        listen 8080;
        location / {
            return 200 "ok\n";
        }
        location /slow {
            proxy_pass http://127.0.0.1:9000;   # a backend that takes 4 seconds
        }
    }
}

run.sh starts a tiny python3 backend (slow.py) that answers after 4 seconds and does two things: ① opens four connections in order 1→4 with bash's /dev/tcp without sending anything, then makes a new connection with curl; ② starts two /slow requests and, while they wait, makes a new connection with curl. In between it briefly attaches gdb to the worker and prints ngx_cycle->connection_n, free_connection_n and reusable_connections_n.

=== idle worker: connection_n, free_connection_n, reusable_connections_n ===
$1 = 6
$2 = 4
$3 = 0
=== 1) four idle connections (opened in order 1..4), then a new one ===
$1 = 6
$2 = 0
$3 = 4
new connection: HTTP 200
$1 = 6
$2 = 1
$3 = 3
idle1: closed by server
idle2: still open
idle3: still open
idle4: still open
=== 2) two requests waiting for the slow backend, then a new one ===
$1 = 6
$2 = 0
$3 = 0
new connection: HTTP 000
curl exit code: 56
slow2: HTTP 200
slow1: HTTP 200
=== error.log ===
[warn] 13#13: 6 worker_connections are not enough, reusing connections
[alert] 13#13: 6 worker_connections are not enough

1. The worker is idle, yet there are 4 free slots, not 6. The listening socket and the channel to the master take one slot each.
2. Four idle connections bring the free count to 0, and all four are reusable. ngx_http_init_connection() marks a connection waiting for its request with ngx_reusable_connection(c, 1) (src/http/ngx_http_request.c line 367).
3. The new connection got a 200, and in exchange idle1 — the first one opened — was closed. ngx_drain_connections() logged the warn (lines 1420–1423) and closed the connection at the back of the list. The new connection's curl had finished and returned its slot, hence 1 free.
4. The two requests waiting for the backend use four slots (2 client + 2 upstream) and are not reusable. With nothing to close, ngx_get_connection() logs the alert and returns NULL. ngx_event_accept() simply closes the socket it already got from accept4() (src/event/ngx_event_accept.c lines 144–151), so curl is cut off without a response. curl's exit code varied between runs: 56 (failure while receiving) or 52 (empty reply).
5. The two waiting requests got their 200 normally after 4 seconds. The pid (13) differs on every run.

Easy to get wrong

• worker_connections is per worker. If connections pile onto one worker (section 7), that worker can run short even while another has slots to spare.
• Connection slots and the file-descriptor limit are separate things. Even with free slots, if the process hits its open-file limit (ulimit -n, worker_rlimit_nofile), accept4() fails with EMFILE. That is why nginx warns at startup when worker_connections exceeds that limit (src/event/ngx_event.c lines 520–531).

4. Registering with epoll and waking up — EPOLLET and the instance bit

Registering: hand the kernel a connection address plus one bit

    if (e->active) {
        op = EPOLL_CTL_MOD;
        events |= prev;

    } else {
        op = EPOLL_CTL_ADD;
    }

#if (NGX_HAVE_EPOLLEXCLUSIVE && NGX_HAVE_EPOLLRDHUP)
    if (flags & NGX_EXCLUSIVE_EVENT) {
        events &= ~EPOLLRDHUP;
    }
#endif

    ee.events = events | (uint32_t) flags;
    ee.data.ptr = (void *) ((uintptr_t) c | ev->instance);

src/event/modules/ngx_epoll_module.c lines 606–621

epoll keeps only one registration per fd, so when adding the read event while the write event is already registered (e->active), it merges the two with EPOLL_CTL_MOD. And in the user data it hands the kernel, data.ptr, it stores the connection struct's address ORed with the event's instance bit. The struct address is aligned, so its lowest bit is always 0 — nginx uses that spot as a one-bit store.

flags decides the trigger mode. In the epoll build NGX_CLEAR_EVENT is EPOLLET (src/event/ngx_event.h line 331).

• Connection sockets are registered by HTTP via ngx_handle_read_event(rev, 0) (src/http/ngx_http_request.c line 369). With an edge-triggered method it adds, with NGX_CLEAR_EVENT, only events that are neither registered (!active) nor ready (!ready) (src/event/ngx_event.c lines 282–294). Once added, the registration stays until the connection closes.
• Listening sockets are added without EPOLLET, as in ngx_add_event(rev, NGX_READ_EVENT, 0) — that is, level-triggered (src/event/ngx_event.c line 909; with several workers section 7's EPOLLEXCLUSIVE is added, but still no EPOLLET). If connections remain in the backlog, the next epoll_wait reports it again, so taking one at a time loses nothing.

With edge triggering there is only one notification, so the receiver must read until the socket is drained (NGX_USE_GREEDY_EVENT). Calling recv() one extra time every time just to see EAGAIN would add system calls, so on kernels where EPOLLRDHUP is available nginx treats a short read as "drained" and clears ready:

            if ((ngx_event_flags & NGX_USE_EPOLL_EVENT)
                && ngx_use_epoll_rdhup)
            {
                if ((size_t) n < size) {
                    if (!rev->pending_eof) {
                        rev->ready = 0;
                    }

                    rev->available = 0;
                }

                return n;
            }

src/os/unix/ngx_recv.c lines 157–169

That is why, in example 2's strace in section 5, recvfrom(3, …, 1024, …) = 27 is not followed by a second recvfrom checking for EAGAIN.

Waking up: ngx_epoll_process_events()

As soon as epoll_wait(ep, event_list, (int) nevents, timer) returns, nginx refreshes its cached time with ngx_time_update() (with timer_resolution, only when a SIGALRM has arrived — src/event/modules/ngx_epoll_module.c lines 800–806). nginx does not read the clock every time it needs the time; it uses ngx_current_msec (milliseconds, based on CLOCK_MONOTONIC) and the time strings for logs and HTTP headers that this function fills once per turn (src/core/ngx_times.c lines 94–107). All handlers in one turn see the same "now" — and if a handler takes long, that "now" lags behind reality.

If epoll_wait fails with EINTR (interrupted by a signal), nginx logs epoll_wait() failed (4: Interrupted system call) at info level and returns (lines 808–824). I saw this line while polishing the examples, whenever I attached gdb to the worker. Next comes the loop over the events (the write side is elided):

    for (i = 0; i < events; i++) {
        c = event_list[i].data.ptr;

        instance = (uintptr_t) c & 1;
        c = (ngx_connection_t *) ((uintptr_t) c & (uintptr_t) ~1);

        rev = c->read;

        if (c->fd == -1 || rev->instance != instance) {

            /*
             * the stale event from a file descriptor
             * that was just closed in this iteration
             */

            ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                           "epoll: stale event %p", c);
            continue;
        }

        revents = event_list[i].events;

        /* ... debug log; on EPOLLERR|EPOLLHUP wake both read and write ... */

        if ((revents & EPOLLIN) && rev->active) {

            /* ... on EPOLLRDHUP, pending_eof = 1 ... */

            rev->ready = 1;
            rev->available = -1;

            if (flags & NGX_POST_EVENTS) {
                queue = rev->accept ? &ngx_posted_accept_events
                                    : &ngx_posted_events;

                ngx_post_event(rev, queue);

            } else {
                rev->handler(rev);
            }
        }

        /* ... the write event (EPOLLOUT) has the same shape: wev->ready = 1, wev->handler(wev) ... */
    }

src/event/modules/ngx_epoll_module.c lines 836–933

• It splits data.ptr back into the low bit (instance) and the connection address.
• It sets ready = 1 on the ready event and calls the handler right there. For a listening socket that is ngx_event_accept; for an HTTP connection, the function that reads the request.
• With NGX_POST_EVENTS (a turn holding the accept mutex, section 7) it queues instead of calling, using the rev->accept bit to choose the accept queue or the normal queue.

The instance bit: a slot closed and reused within the same turn

Why isn't the c->fd == -1 check enough? epoll_wait returns many events at once in one array, and the worker handles them front to back. In between, a slot can change owners.

The free list lends from the head, so a slot just closed goes straight to the next client. The kernel also hands out the lowest free fd number again, so comparing fd numbers can't tell them apart either.

Because ngx_get_connection() flips instance every time it lends a slot, a slot that has been reused carries a different bit from the one the kernel was holding. A slot that was closed and not yet reused is caught by c->fd == -1. The source comment is just the one line "the stale event from a file descriptor that was just closed in this iteration"; the scenario in the picture is my reading of the code (I did not trigger it in practice).

5. Timers — the red-black tree becomes epoll_wait's timeout

client_header_timeout, client_body_timeout, send_timeout, keepalive_timeout, proxy_read_timeout … the many timeout directives are all built on one mechanism: setting a timer on an event. A timer of client_header_timeout is set on the connection's read event; if data arrives first the timer is removed, and if not, the same handler is called with timedout = 1. The handler checks if (rev->timedout) first and closes the connection.

Setting: changes within 300ms are ignored

    key = ngx_current_msec + timer;

    if (ev->timer_set) {

        /*
         * Use a previous timer value if difference between it and a new
         * value is less than NGX_TIMER_LAZY_DELAY milliseconds: this allows
         * to minimize the rbtree operations for fast connections.
         */

        diff = (ngx_msec_int_t) (key - ev->timer.key);

        if (ngx_abs(diff) < NGX_TIMER_LAZY_DELAY) {
            /* ... debug log ... */
            return;
        }

        ngx_del_timer(ev);
    }

    ev->timer.key = key;

    /* ... "event timer add: ..." debug log ... */

    ngx_rbtree_insert(&ngx_event_timer_rbtree, &ev->timer);

    ev->timer_set = 1;

src/event/ngx_event_timer.h lines 56–86

The key is the "expiry time" (ngx_current_msec + timer), and the node ev->timer embedded in the event goes into the tree. When updating an already set timer, if the difference is under NGX_TIMER_LAZY_DELAY (300ms, line 19) the tree is left alone. Fast connections trading small pieces of data reset their timers constantly, and this saves a remove-and-insert each time. It also means a timeout can be off by up to 300ms.

The insert function ngx_rbtree_insert_timer_value() compares keys by subtracting and looking at the sign, as in (ngx_rbtree_key_int_t) (node->key - temp->key) < 0. 32-bit milliseconds wrap to 0 every 49 days, but timers sit within minutes of each other, so subtraction-based comparison survives the wrap (src/core/ngx_rbtree.c lines 129–139).

Finding the nearest, firing the expired

    if (ngx_event_timer_rbtree.root == &ngx_event_timer_sentinel) {
        return NGX_TIMER_INFINITE;
    }

    root = ngx_event_timer_rbtree.root;
    sentinel = ngx_event_timer_rbtree.sentinel;

    node = ngx_rbtree_min(root, sentinel);

    timer = (ngx_msec_int_t) (node->key - ngx_current_msec);

    return (ngx_msec_t) (timer > 0 ? timer : 0);

src/event/ngx_event_timer.c lines 38–49

ngx_event_find_timer() returns NGX_TIMER_INFINITE ((ngx_msec_t) -1, ngx_event_timer.h line 17) when the tree is empty, and that becomes epoll_wait's last argument -1 (wait forever). Otherwise it returns the time left until the leftmost (minimum) node.

    for ( ;; ) {
        root = ngx_event_timer_rbtree.root;

        if (root == sentinel) {
            return;
        }

        node = ngx_rbtree_min(root, sentinel);

        /* node->key > ngx_current_msec */

        if ((ngx_msec_int_t) (node->key - ngx_current_msec) > 0) {
            return;
        }

        ev = ngx_rbtree_data(node, ngx_event_t, timer);

        /* ... "event timer del: ..." debug log ... */

        ngx_rbtree_delete(&ngx_event_timer_rbtree, &ev->timer);

        /* ... */

        ev->timer_set = 0;

        ev->timedout = 1;

        ev->handler(ev);
    }

src/event/ngx_event_timer.c lines 61–95

ngx_event_expire_timers() stops when the minimum is still in the future; otherwise it removes it and calls the handler with timedout = 1, over and over. There is no separate timer handler — the same event handler is called, and it uses timedout to tell "data arrived" from "time ran out".

That this step comes after epoll event handling in section 1 matters too. If data arrives just before a timeout, the data handler runs first and gets the chance to remove or reset the timer.

Example 2: epoll_wait's timeout follows the timers

What Part 1 showed with the defaults (-1 → 60000 → 75000), we now look at with conspicuous values, alongside the timer tree. The example folder is ex/u07-epoll-timers.

# An example written for this post (not part of nginx's source)
# One worker, debug log. Two timers (waiting for request headers 7s, keep-alive 3s) set to conspicuous values
worker_processes 1;
error_log /tmp/u07/error.log debug;
pid /tmp/u07/nginx.pid;

events {
    worker_connections 64;
}

http {
    access_log off;
    client_header_timeout 7s;   # how long to wait for the full request header after connecting
    keepalive_timeout 3s;       # how long to wait for the next request after a response

    server {
        listen 8080;
        location / {
            return 200 "hello\n";
        }
    }
}

With strace -tt -p <worker pid> -e trace=epoll_wait,epoll_ctl,accept4,recvfrom,writev,close attached to the idle worker, nc connects, sends a request 1 second later, and keeps the connection open for 5 more seconds:

(sleep 1; printf 'GET / HTTP/1.1\r\nHost: x\r\n\r\n'; sleep 5) | nc 127.0.0.1 8080

=== strace (worker 11) ===
03:22:08.624393 epoll_wait(7, [{events=EPOLLIN, data={u32=3852001504, u64=94506017415392}}], 512, -1) = 1
03:22:10.039805 accept4(5, {sa_family=AF_INET, sin_port=htons(47908), sin_addr=inet_addr("127.0.0.1")}, [112 => 16], SOCK_NONBLOCK) = 3
03:22:10.066303 epoll_ctl(7, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLRDHUP|EPOLLET, data={u32=3852001985, u64=94506017415873}}) = 0
03:22:10.067509 epoll_wait(7, [{events=EPOLLIN, data={u32=3852001985, u64=94506017415873}}], 512, 7000) = 1
03:22:10.802872 recvfrom(3, "GET / HTTP/1.1\r\nHost: x\r\n\r\n", 1024, 0, NULL, NULL) = 27
03:22:10.810558 writev(3, [{iov_base="HTTP/1.1 200 OK\r\nServer: nginx/1"..., iov_len=147}, {iov_base="hello\n", iov_len=6}], 2) = 153
03:22:10.815907 epoll_wait(7, [], 512, 3000) = 0
03:22:13.820774 close(3)                = 0
03:22:13.822107 epoll_wait(7,  <detached ...>

The timer-related lines from the same run's debug log (date and time columns removed):

[debug] 11#11: timer delta: 4182
[debug] 11#11: epoll timer: -1
[debug] 11#11: accept on 0.0.0.0:8080, ready: 0
[debug] 11#11: *2 accept: 127.0.0.1:47908 fd:3
[debug] 11#11: *2 event timer add: 3: 7000:4051177
[debug] 11#11: timer delta: 1416
[debug] 11#11: epoll timer: 7000
[debug] 11#11: *2 event timer del: 3: 4051177
[debug] 11#11: *2 set http keepalive handler
[debug] 11#11: *2 event timer add: 3: 3000:4047939
[debug] 11#11: timer delta: 762
[debug] 11#11: epoll timer: 3000
[debug] 11#11: timer delta: 3019
[debug] 11#11: *2 event timer del: 3: 4047939
[debug] 11#11: *2 http keepalive handler
[debug] 11#11: *2 close http connection: 3
[debug] 11#11: epoll timer: -1
[debug] 11#11: timer delta: 2488
[debug] 11#11: epoll timer: -1

1. An idle worker sleeps with -1. There are no timers (epoll timer: -1 is printed at epoll module lines 797–798). The first epoll_wait took a while because it waited from strace attaching until the client connected.
2. The listening socket (fd 5) wakes up and accept4(…, SOCK_NONBLOCK) takes the connection — non-blocking from the moment it is accepted.
3. HTTP sets a timer — event timer add: 3: 7000:4051177 (fd 3, 7000ms, expiry-time key). That is client_header_timeout (src/http/ngx_http_request.c line 366). Then epoll_ctl(… EPOLLIN|EPOLLRDHUP|EPOLLET …) — the edge-triggered registration from section 4.
4. The next epoll_wait timeout is 7000, because the tree's minimum is the timer just set.
5. When the request arrives a second later, nginx reads it (one recvfrom; 27 < 1024 so no more reads), answers with writev, removes the header timer (event timer del) and sets the 3000ms keep-alive timer (src/http/ngx_http_request.c line 3571). So the next epoll_wait uses 3000.
6. Nothing happens for 3 seconds, epoll_wait returns = 0, and in the same turn ngx_event_expire_timers() removes the timer and calls the keep-alive handler with timedout = 1. The handler closes the connection (close(3)). timer delta: 3019 is those 3 seconds.
7. The tree is empty again and we are back to -1.

epoll_wait's last argument follows the nearest timer in the tree. The values are the actual strace and debug log output above.

The data.u64 values show section 4's story. The listening socket's value 94506017415392 is even (instance 0) and the new connection's 94506017415873 is odd (instance 1). At the end of the same run I looked into the worker with gdb:

=== gdb: where do the epoll data pointers point? ===
$1 = 240
$2 = (ngx_connection_t *) 0x55f3e598e0e0
$3 = 5
$4 = 1

sizeof(ngx_connection_t) is 240, the connection array starts at 0x55f3e598e0e0 (= 94506017415392), and connections[0].fd is 5 — the listening socket is slot 0. The new connection's value minus 1 is start + 480 = slot 2 (slot 1 is the channel to the master), and connections[2].read->instance is 1. Slot 2 had been used and returned once by the startup check's curl (flipping 1 → 0), and lending it again flipped it 0 → 1. Addresses, times, ports and pids differ on every run.

Easy to get wrong

• Timers are per worker. The tree is a single global ngx_event_timer_rbtree, but each process has its own memory, so workers' timers never mix.
• Graceful shutdown looks at the timer tree. When ngx_exiting is set in the worker loop, ngx_event_no_timers_left() decides whether to exit by checking "are any non-cancelable timers left?" (src/event/ngx_event_timer.c lines 99–126). Exactly as seen in Part 6.

6. Posted events — not now, at the end of this turn

Calling another event's handler directly from inside a handler deepens the call stack, and the callee may touch state the caller is still cleaning up. So nginx expresses "call it a bit later" with queues.

#define ngx_post_event(ev, q)                                                 \
                                                                              \
    if (!(ev)->posted) {                                                      \
        (ev)->posted = 1;                                                     \
        ngx_queue_insert_tail(q, &(ev)->queue);                               \
                                                                              \
        ngx_log_debug1(NGX_LOG_DEBUG_CORE, (ev)->log, 0, "post event %p", ev);\
                                                                              \
    } else  {                                                                 \
        ngx_log_debug1(NGX_LOG_DEBUG_CORE, (ev)->log, 0,                      \
                       "update posted event %p", ev);                         \
    }

src/event/ngx_event_posted.h lines 17–28

An event already on a queue (posted) is not added twice. ngx_event_process_posted(), which drains a queue, just loops while (!ngx_queue_empty(posted)), taking from the front and doing ngx_delete_posted_event(ev); ev->handler(ev); (src/event/ngx_event_posted.c lines 18–36). Anything added to the same queue during processing is handled within that same call. There are three queues:

The three queues from the table above, placed on section 1's single iteration. The order is from ngx_process_events_and_timers() in ngx_event.c; the queue is chosen in ngx_epoll_module.c lines 894–931.

ngx_posted_next_events is a fairness device. Sending one huge response to the end in a single turn would make every other connection wait, so when the write filter stops sending with data still left, it puts the write event on this queue:

    if (chain && c->write->ready && !c->write->delayed) {
        ngx_post_event(c->write, &ngx_posted_next_events);
    }

src/http/ngx_http_write_filter_module.c lines 334–336

At the start of the next turn, ngx_event_move_posted_next() sets ready = 1 again on those events and moves the whole queue into ngx_posted_events (src/event/ngx_event_posted.c lines 39–60). That turn's epoll_wait(…, 0) first collects and handles other connections' events, and the rest of the response is sent at the end. When and how much of a response is sent is covered in Part 10.

Easy to get wrong: edge triggering and the posted queue

Edge triggering notifies only "once, when it becomes readable", so if data has already arrived when a handler is swapped, epoll won't tell you again. That is why HTTP code has if (rev->ready) { ngx_post_event(rev, &ngx_posted_events); } in many places — for example when the next request is already waiting right after switching to keep-alive (src/http/ngx_http_request.c lines 3573–3575). nginx queues for itself what epoll will not report again.

7. accept — with several workers, who takes the new connection?

The accept handler

The read-event handler of a listening socket is ngx_event_accept():

    if (!(ngx_event_flags & NGX_USE_KQUEUE_EVENT)) {
        ev->available = ecf->multi_accept;
    }

    /* ... */

    do {
        socklen = sizeof(ngx_sockaddr_t);

#if (NGX_HAVE_ACCEPT4)
        if (use_accept4) {
            s = accept4(lc->fd, &sa.sockaddr, &socklen, SOCK_NONBLOCK);
        } else {
            s = accept(lc->fd, &sa.sockaddr, &socklen);
        }
#else
        s = accept(lc->fd, &sa.sockaddr, &socklen);
#endif

        /* ... failure handling: return on EAGAIN, briefly disable accept on EMFILE ... */

        ngx_accept_disabled = ngx_cycle->connection_n / 8
                              - ngx_cycle->free_connection_n;

        c = ngx_get_connection(s, ev->log);

        /* ... create the connection pool, copy the address, set recv/send, number the connection ... */

        ls->handler(c);

        /* ... */

    } while (ev->available);

src/event/ngx_event_accept.c lines 47–318

• With multi_accept off (the default), available = 0, so the loop runs once and takes one connection. The listening socket is level-triggered, so remaining connections are reported again next turn. With on it accepts until EAGAIN.
• ngx_accept_disabled becomes positive when free slots are under 1/8 of the total. It marks "this worker is fairly full" and is used by the accept mutex and the EPOLLEXCLUSIVE re-add below.
• ls->handler is the function set by the module that created the listening socket. For HTTP it is ngx_http_init_connection (src/http/ngx_http.c line 1882). From here on it is Part 8.

In the epoll build the accepted connection is not added to epoll here. ngx_add_conn is called only without NGX_USE_EPOLL_EVENT (lines 302–307); the epoll registration happens when HTTP needs the read event (section 4). That is why example 2's strace showed a single epoll_ctl after accept4.

One listening socket, several workers — three paths

The listen 8080; socket is opened by the master, and the workers inherit the same socket through fork() (Part 6). Wake every worker for one client and only one accept4() succeeds while the rest come back empty with EAGAIN (often called the "thundering herd"). The part of this version's ngx_event_process_init() that adds the listening socket to epoll has three paths:

#if (NGX_HAVE_REUSEPORT)

        if (ls[i].reuseport) {
            if (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {
                return NGX_ERROR;
            }

            continue;
        }

#endif

        if (ngx_use_accept_mutex) {
            continue;
        }

#if (NGX_HAVE_EPOLLEXCLUSIVE)

        if ((ngx_event_flags & NGX_USE_EPOLL_EVENT)
            && ccf->worker_processes > 1)
        {
            ngx_use_exclusive_accept = 1;

            if (ngx_add_event(rev, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
                == NGX_ERROR)
            {
                return NGX_ERROR;
            }

            continue;
        }

#endif

        if (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {
            return NGX_ERROR;
        }

src/event/ngx_event.c lines 875–911

The yellow path is the older approach using a lock in shared memory; the two blue paths leave the job to the kernel.

(a) The default — accept_mutex off, epoll, two or more workers — registers the listening socket with EPOLLEXCLUSIVE, a Linux 4.5 flag that wakes only some of the epoll instances watching the same fd (this description comes from the Linux documentation; I did not verify it in kernel source). It arrived in commit 5c2dd3913 (2016-07-15, from 1.11.3), the same day the accept_mutex default was turned off. But the kernel usually wakes the epoll that registered first, piling connections onto one worker, so commit 96c342e56 (2021-12-30, "Events: fixed balancing between workers with EPOLLEXCLUSIVE.", from 1.21.6) added ngx_reorder_accept_events(). Called at the end of the accept handler, it borrows the requests field of the listening socket's slot as a counter:

    if (c->requests++ % 16 != 0
        && ngx_accept_disabled <= 0)
    {
        return;
    }

    if (ngx_del_event(c->read, NGX_READ_EVENT, NGX_DISABLE_EVENT)
        == NGX_ERROR)
    {
        return;
    }

    if (ngx_add_event(c->read, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
        == NGX_ERROR)
    {
        return;
    }

src/event/ngx_event_accept.c lines 458–474

Once every 16 calls (or every time, if this worker is fairly full) it removes the listening socket from epoll and adds it back, moving to the back of the line.

(b) With accept_mutex on the listening socket is not added to epoll at first (the continue above). Instead, at step 2 of every turn, ngx_trylock_accept_mutex() tries a lock in shared memory (src/event/ngx_event_accept.c lines 326–361). The lock is a single atomic compare-and-swap: *mtx->lock == 0 && ngx_atomic_cmp_set(mtx->lock, 0, ngx_pid) (src/core/ngx_shmtx.c line 65). The winner puts the listening socket into epoll with ngx_enable_accept_events(); a loser that used to hold it removes it with ngx_disable_accept_events() and tries again within accept_mutex_delay (500ms). A turn holding the lock defers every event with NGX_POST_EVENTS, and right after epoll_wait it handles only the accept queue and then releases the lock immediately (steps 5 and 6 in section 1). Holding the lock while handling ordinary requests would keep the other workers from accepting during that time. A worker with ngx_accept_disabled > 0 skips lock attempts for a few turns (the ngx_accept_disabled-- in section 1's code), yielding to less loaded workers. This path is enabled only with master_process on and two or more workers (src/event/ngx_event.c lines 645–652).

(c) With listen 8080 reuseport, ngx_clone_listening() duplicates the listening-socket definition once per worker while the config is read, giving each a worker number (src/core/ngx_connection.c lines 99–131). Each socket binds the same port with SO_REUSEPORT, and each worker skips sockets with ls[i].worker != ngx_worker (lines 804–806), adding only its own to epoll. The kernel picks the socket for each connection. No lock, no re-adding.

Example 3: two workers, 300 connections — who took them?

The example folder is ex/u07-accept. From one config template (nginx.conf.in), @ACCEPT_MUTEX@ and @LISTEN@ are substituted to start three variants in turn; each time 300 curl requests are sent (a new connection per request, 10 at a time), and the $pid written to the access log counts how many each worker took. The heart of the config is these lines:

worker_processes 2;
error_log /tmp/u07/error.log @LOGLEVEL@;

events {
    worker_connections 64;
    accept_mutex @ACCEPT_MUTEX@;
}

http {
    # record which worker took the request ($pid = pid of the worker that handled it)
    log_format w '$pid';
    access_log /tmp/u07/access.log w;

    server {
        listen @LISTEN@;

(The "example written for this post" comment on the first line, pid and the location are trimmed.) From the debug log (@LOGLEVEL@ = debug), I kept the epoll add event lines without EPOLLET (80000000) — the registrations of the listening socket and the channel — taking only the first per worker and fd, and annotated what each fd is using /proc/<pid>/fd and /proc/net/tcp. The line is printed at epoll module lines 623–625.

=== (a) default: accept_mutex off ===
-- epoll_ctl without EPOLLET (first per worker and fd):
[debug] 12#12: epoll add event: fd:6 op:1 ev:10000001   <- listening socket :8080
[debug] 12#12: epoll add event: fd:7 op:1 ev:00002001   <- channel to master
[debug] 13#13: epoll add event: fd:6 op:1 ev:10000001   <- listening socket :8080
[debug] 13#13: epoll add event: fd:9 op:1 ev:00002001   <- channel to master
-- requests per worker:
    154 12
    146 13
-- debug log counts:
    22  epoll add event: fd:[0-9]* op:[0-9]* ev:1
     0  accept mutex locked
     0  accept mutex lock failed
     0  post event
=== (b) accept_mutex on ===
-- epoll_ctl without EPOLLET (first per worker and fd):
[debug] 366#366: epoll add event: fd:7 op:1 ev:00002001   <- channel to master
[debug] 366#366: epoll add event: fd:6 op:1 ev:00002001   <- listening socket :8080
[debug] 367#367: epoll add event: fd:9 op:1 ev:00002001   <- channel to master
[debug] 367#367: epoll add event: fd:6 op:1 ev:00002001   <- listening socket :8080
-- requests per worker:
    104 366
    196 367
-- debug log counts:
     0  epoll add event: fd:[0-9]* op:[0-9]* ev:1
   512  accept mutex locked
     6  accept mutex lock failed
   603  post event
=== (c) listen 8080 reuseport ===
-- epoll_ctl without EPOLLET (first per worker and fd):
[debug] 720#720: epoll add event: fd:6 op:1 ev:00002001   <- listening socket :8080
[debug] 720#720: epoll add event: fd:8 op:1 ev:00002001   <- channel to master
[debug] 721#721: epoll add event: fd:7 op:1 ev:00002001   <- listening socket :8080
[debug] 721#721: epoll add event: fd:10 op:1 ev:00002001   <- channel to master
-- requests per worker:
    141 720
    159 721
-- debug log counts:
     0  epoll add event: fd:[0-9]* op:[0-9]* ev:1
     0  accept mutex locked
     0  accept mutex lock failed
     0  post event

ev: is the event mask passed to epoll, in hex: 00000001 = EPOLLIN, 00002000 = EPOLLRDHUP, 10000000 = EPOLLEXCLUSIVE, 80000000 = EPOLLET.

1. (a) Both workers registered the same fd 6 as 10000001 (EPOLLIN|EPOLLEXCLUSIVE). EPOLLRDHUP is missing because ngx_epoll_add_event() clears it for exclusive registrations (lines 614–618 in section 4). The order matches the source too: the listening socket (init_process) before the channel (ngx_add_channel_event). There are 22 registrations starting with ev:1… — the first two plus a re-add every 16 of some 300 accepts. The split is 154 : 146.
2. (b) The listening socket is registered after the channel, without exclusive (00002001). It is not added at startup but in the first turn that wins the mutex. "accept mutex locked" appears 512 times — a lock attempt every turn. Since every event is deferred, "post event" appears 603 times. The split leaned to 104 : 196. In runs while polishing the example it leaned even more, like 226 : 74 and 274 : 26, and a re-run for checking even gave 300 : 0 (one worker took everything); which side it leaned to varied between runs.
3. (c) The two workers' listening-socket fds differ, 6 and 7. Each took one separate socket. The split is 141 : 159.

Pids and counts differ on every run. The connections are extremely short and the curl clients run on the same machine, so this says nothing about distribution in a real service. What the source tells us is only that the three paths divide the work with different mechanisms.

Easy to get wrong

• Old tuning articles telling you to "turn accept_mutex off" describe what is already the default. Before 2016 (1.11.3) the default was on. Part 1 pointed this out as well.
• With worker_processes 1 none of the three paths matters. The accept mutex is only enabled with two or more workers (line 645), and EPOLLEXCLUSIVE also requires worker_processes > 1 (lines 893–894). In example 2's single-worker config, the listening socket is registered as a plain level-triggered event.

What this post leaves out

• The other event-method modules in src/event/modules/: ngx_kqueue_module.c (BSD, macOS), ngx_poll_module.c, ngx_select_module.c, ngx_devpoll_module.c, ngx_eventport_module.c, ngx_iocp_module.c (Windows). The only event-method module in this build's module list (objs/ngx_modules.c) is epoll
• The write side, ngx_handle_write_event() and send_lowat: src/event/ngx_event.c lines 345–425
• eventfd, which tells the loop that thread-pool or file-AIO work is done (ngx_epoll_notify_init·ngx_epoll_notify_handler: src/event/modules/ngx_epoll_module.c lines 383–459; ngx_epoll_notify: line 765; ngx_epoll_eventfd_handler for file AIO: from line 939)
• ngx_epoll_test_rdhup(), which tests EPOLLRDHUP support at startup: lines 464–526 of the same file
• The path that briefly disables accept on EMFILE, and ngx_enable_accept_events/ngx_disable_accept_events: src/event/ngx_event_accept.c lines 364–426
• UDP's ngx_event_recvmsg(): src/event/ngx_event_udp.c
• The non-blocking upstream connect ngx_event_connect_peer(): src/event/ngx_event_connect.c — Part 11
• ngx_close_connection() (remove timers → remove from epoll → remove from posted queues → return the slot → close()): src/core/ngx_connection.c lines 1286–1370

Summary

What's next

We have followed the event loop up to accepting a connection on the listening socket and calling ls->handler(c), that is ngx_http_init_connection(). As example 2 showed, that function sets the client_header_timeout timer, hands the read event to epoll and returns right away. Starting from ngx_http_wait_request_handler(), the handler this loop calls when data arrives, the bytes are parsed into a request line and headers and become an ngx_http_request_t in Part 8: Reading an HTTP Request — From a Connection to ngx_http_request_t.

Other servers solved the same problem differently. In httpd Part 8: MPM, the event MPM's single listener thread watches listening sockets and keep-alive connections in one pollset and hands requests to worker threads. The Poller in Tomcat Part 5: Connector and NIO Endpoint likewise watches many connections with one selector and hands them to a worker thread pool. An nginx worker has no such "hand-off" — the thread that watches is the thread that handles, so there is no queue or lock between threads, but in exchange handlers must never wait. The accept mutex used by httpd's prefork (same Part 8) is the same idea as nginx's accept_mutex, which nginx now leaves off by default in favour of the kernel (EPOLLEXCLUSIVE, SO_REUSEPORT). Reactor Netty, which Spring Part 24: WebFlux runs on by default, is also an event-loop server (Netty's source was not read in this series).

• Part 2: The Big Picture — Master and Workers, Modules, the Cycle and Config Contexts
• Part 3: Memory Pools and Core Data Structures — the red-black tree the timers use
• Part 6: Master and Workers — Processes, Signals and Zero-Downtime Reload
• Part 10: Filters and Static Files — where the posted next queue splits a response