Digging into nginx's Source 6: Master and Workers — Processes, Signals and Zero-Downtime Reload

What this post covers

This is the sixth post in the "Digging into nginx's Source" series. In Part 4 we followed main() as it reads the config file and builds a cycle (the struct that holds everything read from the config) with ngx_init_cycle(), and in Part 5 we saw how one line of config becomes a module's config struct. The last line of main() is ngx_master_process_cycle(cycle). This time we step inside that function.

On the map from Part 2, this post zooms into the Master · workers box in B. Processes and events.

The green box on the Part 2 map is this post. The event loop box next to it (what runs inside a worker) is Part 7.

We want to answer four questions.

• What do the master process and worker process lines in ps each do, and how do they talk to each other?
• When you type nginx -s reload, what happens inside the running nginx, and in what order? Why are requests in flight not cut off?
• When a worker dies, who starts it again, and how?
• How does USR2 swap the running nginx binary for a new one without taking the server down?

All line numbers in this post refer to this commit. How to get the same source and build the Docker image used for the examples is in Part 1.

ngx_process_cycle.c is 1,241 lines long and mixes in things like the cache manager processes and the control API, which is disabled in this build. This post keeps to the main path that a first-time reader can follow; what was left out is listed by name and location in "What this post leaves out" at the end.

Terms to know first

Master process and worker processes. When you start nginx you get one process running as root (the master) and as many child processes (workers) as worker_processes says. Only the workers accept connections and handle requests; the master is the manager that reads the config, opens the listening sockets, and starts and watches the workers. This post checks in the source the structure we saw as a single picture in Part 2.

A signal is a short notification the operating system delivers to a process. It carries only a number; the receiving process pauses what it was doing and runs the signal handler function it registered. kill -HUP 1234 means "send SIGHUP to process 1234". Because a handler can cut in at any point in the program, there is little it can safely do (calling something with internal state, like malloc(), is dangerous). So nginx's handler does little more than set one flag variable to 1.

sig_atomic_t flags. A variable read and written by both a handler and the main code is declared as sig_atomic_t, an integer type that the C standard guarantees will never be seen half-written even if a handler interrupts. nginx's ngx_reconfigure, ngx_quit, ngx_terminate and ngx_reopen are of this type.

sigsuspend() is a system call that does "temporarily unblock the blocked signals, sleep until a signal arrives, then block them again" in one step. The master normally sleeps inside it. Why not just pause()? We'll see in section 2.

The channel is nginx's own line between master and worker. Before forking a worker, socketpair() creates a pair of connected Unix domain sockets; the master uses one end and the worker the other. The master writes 32-byte commands such as "stop accepting and leave (QUIT)" into it.

A graceful shutdown means "accept no new connections, but finish the requests in progress before leaving". That is why reload causes no downtime; in nginx, SIGQUIT (or the channel's NGX_CMD_QUIT) asks for it. "Leave right now" is SIGTERM.

1. The process map — what the master holds, what the workers hold

Yellow boxes are data the master holds; green boxes are the workers that do the actual work.

Example 1: start it and look with ps

The examples in this post were run with nginx 1.31.9 built as-is from the base commit (9a7439fd79), in a Docker container (Debian bookworm, --with-debug). Inside the container we run as root. The config for the first example, ex/u06-reload:

# Example written for this post (not part of nginx's source)
# Workers before and after reload: the response carries a version tag (v1) and the pid of the worker that handled it.
# run.sh copies this file to /tmp/u06/, then changes v1 to v2 and reloads.
worker_processes 2;
error_log /tmp/u06/error.log notice;
pid /tmp/u06/nginx.pid;

events {
    worker_connections 64;
}

http {
    access_log off;
    keepalive_timeout 60s;

    server {
        listen 8080;

        location = /hello {
            return 200 "v1 pid=$pid\n";
        }

        # a backend that answers 3 seconds late — to create a request in progress
        location = /slow {
            proxy_pass http://127.0.0.1:9000;
        }
    }
}

$pid is a variable that returns the pid of the worker process handling the request. Here is ps right after start, plus a first request (the first part of run.sh; the full output continues in section 4).

=== 1. after start ===
    PID    PPID USER     CMD
     19       1 root     nginx: master process nginx -c /tmp/u06/nginx.conf
     20      19 nobody   nginx: worker process
     21      19 nobody   nginx: worker process
 0.55s curl /hello: v1 pid=21

pids and times differ on every run. Three things to read here:

1. The master is root, the workers are nobody. The config has no user, and we started as root, so the default nobody was used (ngx_core_module_init_conf() fills it from NGX_USER; in this build's objs/ngx_auto_config.h, NGX_USER is "nobody"). Sockets only root may open, such as port 80, are opened by the master, while the workers — which deal directly with requests from the internet — give up their privileges. Where they do that is ngx_worker_process_init() in section 6.
2. The workers' parent (PPID) is the master, because the master created them with fork(). The master's parent is process 1 (the shell running run.sh in the container). With daemon on (the default), ngx_daemon() forks the nginx process the shell started once more and the original exits, so the orphaned master is handed to process 1 (src/os/unix/ngx_daemon.c lines 17–27). That is why the first few lines of error.log (using the "epoll" event method and so on) are written by pid 18, and from start worker processes on by the master, 19.
3. The process names have changed to nginx: master process .... ngx_setproctitle() does that. The master appends its command line (master process nginx -c ...), workers say worker process, and a worker in graceful shutdown says worker process is shutting down. You can read the state from ps alone.

ngx_processes[] — the master's ledger

The master remembers its children in the global array ngx_processes[]. One slot looks like this:

typedef void (*ngx_spawn_proc_pt) (ngx_cycle_t *cycle, void *data);

typedef struct {
    ngx_pid_t           pid;
    int                 status;
    ngx_socket_t        channel[2];

    ngx_spawn_proc_pt   proc;
    void               *data;
    char               *name;

    unsigned            respawn:1;
    unsigned            just_spawn:1;
    unsigned            detached:1;
    unsigned            exiting:1;
    unsigned            exited:1;
} ngx_process_t;

src/os/unix/ngx_process.h lines 20–36

The array has NGX_MAX_PROCESSES (1024) slots, and the five values of the respawn argument passed when forking — NGX_PROCESS_NORESPAWN (-1), _JUST_SPAWN (-2), _RESPAWN (-3), _JUST_RESPAWN (-4), _DETACHED (-5) — decide the combination of the bits above (src/os/unix/ngx_process.h lines 47–53; the bits are set in src/os/unix/ngx_process.c lines 220–251).

They are negative for a reason. ngx_spawn_process() reads this argument as "restart in this slot number" when it is 0 or more (section 3). When a dead worker is brought back in the same slot, the slot number is passed as is.

The function that gives birth to workers: ngx_spawn_process()

The master first starts the workers with the cycle it got from main(). ngx_start_worker_processes() logs "start worker processes" and then, n times, calls ngx_spawn_process(cycle, ngx_worker_process_cycle, (void *) (intptr_t) i, "worker process", type) and ngx_pass_open_channel(cycle) — that is all (src/os/unix/ngx_process_cycle.c lines 357–371).

n is the worker_processes value (with auto, ngx_set_worker_processes() stores the CPU count ngx_ncpu, src/core/nginx.c lines 1637–1640). For each worker, ngx_spawn_process() gets the function to run, ngx_worker_process_cycle, and the worker number i. Because it takes a function pointer, the same function also starts the cache manager process (ngx_cache_manager_process_cycle) and the new binary (ngx_execute_proc).

The heart of ngx_spawn_process() is two places: socketpair() and fork().

    if (respawn != NGX_PROCESS_DETACHED) {

        /* Solaris 9 still has no AF_LOCAL */

        if (socketpair(AF_UNIX, SOCK_STREAM, 0, ngx_processes[s].channel) == -1)
        {
            /* ... "socketpair() failed" ... */
            return NGX_INVALID_PID;
        }

        /* ... both ends non-blocking, FIOASYNC and F_SETOWN on channel[0], FD_CLOEXEC on both ... */

        ngx_channel = ngx_processes[s].channel[1];

    } else {
        ngx_processes[s].channel[0] = -1;
        ngx_processes[s].channel[1] = -1;
    }

    ngx_process_slot = s;


    pid = fork();

    switch (pid) {

    case -1:
        /* ... "fork() failed" log, close the channel ... */
        return NGX_INVALID_PID;

    case 0:
        ngx_parent = ngx_pid;
        ngx_pid = ngx_getpid();
        proc(cycle, data);
        break;

    default:
        break;
    }

src/os/unix/ngx_process.c lines 113–204

1. Before fork(), the global ngx_channel gets the child's end (channel[1]) and ngx_process_slot gets the slot number. Since fork() copies all of memory, the child knows "this is my channel, this is my slot" from the moment it is born.
2. The child (case 0) enters proc(cycle, data) — that is, ngx_worker_process_cycle(cycle, (void *) i) — and never returns. A worker's whole life is the infinite loop inside that function.
3. The parent (master) falls through default, writes pid, proc, data and name into the slot, and sets the bits according to the respawn argument (lines 206–251).

ngx_pass_open_channel() sends the new worker's channel[0] to the other children (NGX_CMD_OPEN_CHANNEL, using SCM_RIGHTS, which passes a file descriptor itself over a socket). It is meant to let workers know each other's channels too, but in this build's core the only places that write to a channel (ngx_write_channel()) are three master-side functions in ngx_process_cycle.c (lines 447, 519, 610). We'll see these messages go back and forth in the strace in section 4.

Easy to get wrong: however many workers you run, there is one listening socket

The socket for listen 8080; is opened once by the master when it reads the config (ngx_init_cycle() → ngx_open_listening_sockets(), Part 4), and the workers inherit the same socket through fork(). That is how several workers can accept() on the same port. Which worker gets a new connection (accept_mutex, reuseport) is the topic of Part 7. The master itself never calls accept() on that socket — it has no event loop.

2. Signals become flags, and the master waits in sigsuspend()

signals[] — the signal name table

The signals nginx handles are gathered in one table.

ngx_signal_t  signals[] = {
    { ngx_signal_value(NGX_RECONFIGURE_SIGNAL),
      "SIG" ngx_value(NGX_RECONFIGURE_SIGNAL),
      "reload",
      ngx_signal_handler },

    { ngx_signal_value(NGX_REOPEN_SIGNAL),
      "SIG" ngx_value(NGX_REOPEN_SIGNAL),
      "reopen",
      ngx_signal_handler },

    /* ... NOACCEPT "", TERMINATE "stop", SHUTDOWN "quit", CHANGEBIN "" ...
       ... SIGALRM, SIGINT, SIGIO, SIGCHLD ... */

    { SIGSYS, "SIGSYS, SIG_IGN", "", NULL },

    { SIGPIPE, "SIGPIPE, SIG_IGN", "", NULL },

    { 0, NULL, "", NULL }
};

src/os/unix/ngx_process.c lines 39–83

Each entry has four fields: the signal number, the name to log, the name you write after nginx -s, and the handler. Names like NGX_RECONFIGURE_SIGNAL are defined as letters like HUP and USR1 in ngx_config.h (src/core/ngx_config.h lines 60–71 — on Linux without NGX_LINUXTHREADS, REOPEN is USR1 and CHANGEBIN is USR2). ngx_signal_value(HUP) pastes tokens with ## to make SIGHUP, and ngx_value() turns the macro into a string, so "SIG" "HUP" → "SIGHUP".

In summary:

nginx -s reload boils down to finding "reload" in the third field of the table and kill()ing with that signal (ngx_os_signal_process(), src/os/unix/ngx_process.c lines 631–648). Only stop, quit, reopen and reload are allowed after -s, so USR2 and WINCH must be sent with kill directly (src/core/nginx.c lines 950–957).

The handler only sets a flag; the loop does the real work.

The handler only sets flags

    switch (ngx_process) {

    case NGX_PROCESS_MASTER:
    case NGX_PROCESS_SINGLE:
        switch (signo) {

        case ngx_signal_value(NGX_SHUTDOWN_SIGNAL):
            ngx_quit = 1;
            action = ", shutting down";
            break;

        /* ... TERMINATE/SIGINT → ngx_terminate, NOACCEPT → ngx_noaccept ... */

        case ngx_signal_value(NGX_RECONFIGURE_SIGNAL):
            ngx_reconfigure = 1;
            action = ", reconfiguring";
            break;

        /* ... REOPEN, CHANGEBIN, SIGALRM, SIGIO ... */

        case SIGCHLD:
            ngx_reap = 1;
            break;
        }

        break;

src/os/unix/ngx_process.c lines 340–406

ngx_signal_handler() reads the same signal differently depending on the process type (ngx_process — master, single process, worker, helper). On the worker side (lines 408–441), SIGHUP and SIGUSR2 are ignored with ", ignoring" — reload and binary upgrade are the master's job. At the end it logs one line, "signal %d (%s) received from %P%s" (lines 444–453), and for SIGCHLD it reaps finished children with ngx_process_get_status() (lines 462–464, section 3).

SIGWINCH is accepted only when running as a daemon (if (ngx_daemonized) { ngx_noaccept = 1; ... }, lines 357–362). An nginx started in the foreground may get SIGWINCH every time the terminal window is resized, and having every worker leave each time would be a problem — that is how we read it (the source does not state the reason).

The master loop: block them, and receive only in sigsuspend()

ngx_master_process_cycle() blocks every signal it handles as soon as it starts.

    sigemptyset(&set);
    sigaddset(&set, SIGCHLD);
    sigaddset(&set, SIGALRM);
    /* ... SIGIO, SIGINT, HUP, USR1, WINCH, TERM, QUIT, USR2 ... */

    if (sigprocmask(SIG_BLOCK, &set, NULL) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "sigprocmask() failed");
    }

    sigemptyset(&set);

src/os/unix/ngx_process_cycle.c lines 87–104

A blocked signal is not thrown away; it stays pending. The last line empties set so that the sigsuspend(&set) in the loop means "block nothing while asleep".

    for ( ;; ) {
        /* ... when shutting down, schedule SIGALRM with setitimer() ... */

        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "sigsuspend");

        sigsuspend(&set);

        ngx_time_update();

        ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
                       "wake up, sigio %i", sigio);

        if (ngx_reap) {
            ngx_reap = 0;
            ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "reap children");

            live = ngx_reap_children(cycle);
        }

src/os/unix/ngx_process_cycle.c lines 139–175

Why go to this trouble? If you leave signals unblocked and write "check the flags → if nothing is set, sleep in pause()", a signal can arrive right after the check and right before sleeping. The handler sets the flag and returns, but the loop has already checked, so it goes to sleep without seeing the flag, and the reload waits until the next signal. If signals are normally blocked, one that arrives in that gap stays pending and is delivered the moment sigsuspend() unblocks. "Unblock and sleep" happens inside one system call, so there is no gap.

So the master's day is "sleep → wake on a signal → check the flags in order → sleep again". The order in which the flags are checked is the priority.

Red boxes are the shutdown side. While shutting down, the loop goes back to the top without looking at other flags such as reload.

The shutdown code (src/os/unix/ngx_process_cycle.c lines 199–231) has two branches.

• nginx -s stop (ngx_terminate): sends TERM to the workers, and at the top of the loop (lines 140–159) schedules a SIGALRM delay milliseconds later with setitimer(). Each time the alarm fires, delay doubles (50 → 100 → 200 …), and once it exceeds 1000 the signal becomes SIGKILL. Even a worker that won't listen ends eventually.
• nginx -s quit (ngx_quit): sends QUIT to the workers and closes the master's own listening sockets too. When all workers have finished and live becomes 0, ngx_master_process_exit() at the top deletes the pid file and exits.

Easy to get wrong: workers do not block signals

Unlike the master, a worker unblocks all signals in ngx_worker_process_init() (sigprocmask(SIG_SETMASK, &set, NULL), lines 924–929). A worker waits in epoll_wait(); when a signal arrives, epoll_wait() returns with EINTR and the loop checks the flags. The event loop plays the role sigsuspend() plays for the master. The exact order of one turn of the event loop is in Part 7.

3. When a worker dies — SIGCHLD and ngx_reap_children()

Reaping finished children

When a child process ends, the kernel sends SIGCHLD to its parent. Inside the handler, ngx_process_get_status() repeats waitpid(-1, &status, WNOHANG) until it returns 0, reaping every finished child and writing exited = 1 in its slot (lines 482–531). Then it logs why the child ended.

        if (WEXITSTATUS(status) == 2 && ngx_processes[i].respawn) {
            ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, 0,
                          "%s %P exited with fatal code %d "
                          "and cannot be respawned",
                          process, pid, WEXITSTATUS(status));
            ngx_processes[i].respawn = 0;
        }

        ngx_unlock_mutexes(pid);

src/os/unix/ngx_process.c lines 551–559

Exit code 2 is an agreement that "restarting won't help". If setuid() or a module's init_process fails during worker setup, the worker ends with exit(2) (ngx_worker_process_init() has several /* fatal */ exit(2);), and restarting such a worker would repeat the same failure forever, so respawn is turned off. ngx_unlock_mutexes() force-releases shared memory locks the dead process held (accept_mutex, the slab lock of shared memory zones). If a worker dies holding a lock, the others would wait forever.

Once the handler sets ngx_reap = 1, the master loop calls ngx_reap_children(). Here is the restart part:

            if (ngx_processes[i].respawn
                && !ngx_processes[i].exiting
                && !ngx_terminate
                && !ngx_quit)
            {
                if (ngx_spawn_process(cycle, ngx_processes[i].proc,
                                      ngx_processes[i].data,
                                      ngx_processes[i].name, i)
                    == NGX_INVALID_PID)
                {
                    ngx_log_error(NGX_LOG_ALERT, cycle->log, 0,
                                  "could not respawn %s",
                                  ngx_processes[i].name);
                    continue;
                }


                ngx_pass_open_channel(cycle);

                live = 1;

                continue;
            }

src/os/unix/ngx_process_cycle.c lines 615–637

All four conditions must hold: it is a child to restart (respawn), the master never told it to leave (!exiting), and the master itself is not shutting down (terminate, quit). It passes the recorded proc, data and name as is, and as the last argument the slot number i (the "0 or more means this slot" from section 1). So the revived worker has the same worker number (the i in data) as the dead one. If you pinned workers to CPUs with worker_cpu_affinity, the revived worker goes to the same CPU.

This function also closes the dead child's channel and sends NGX_CMD_CLOSE_CHANNEL to the other children so they forget it (lines 585–613). It returns live = 1 if at least one child is alive (detached children are not counted).

Example 2: kill -9 a worker

ex/u06-respawn starts a similar config (worker_processes 2) with error_log ... debug; and kill -9s one worker. error.log is shown without the debug lines.

=== 1. kill -9 one worker ===
    PID    PPID USER     CMD
     11       1 root     nginx: master process nginx -c /tmp/u06r/nginx.conf
     12      11 nobody   nginx: worker process
     13      11 nobody   nginx: worker process
--- kill -9 12
    PID    PPID USER     CMD
     11       1 root     nginx: master process nginx -c /tmp/u06r/nginx.conf
     13      11 nobody   nginx: worker process
     22      11 nobody   nginx: worker process
--- error.log (non-debug lines)
2026/11/07 02:51:03 [notice] 11#11: signal 17 (SIGCHLD) received from 12
2026/11/07 02:51:03 [alert] 11#11: worker process 12 exited on signal 9
2026/11/07 02:51:03 [notice] 11#11: start worker process 22
2026/11/07 02:51:03 [notice] 11#11: signal 29 (SIGIO) received

pids and times differ on every run. Within 0.5 seconds worker 12 is gone and 22 has appeared. The three log lines match three functions: signal 17 (SIGCHLD) received from 12 is ngx_signal_handler(), exited on signal 9 is the WTERMSIG branch of ngx_process_get_status() (lines 533–543, alert because it died from a signal), and start worker process 22 is line 206 of the ngx_spawn_process() that ngx_reap_children() called.

The three dark boxes below kill -9 12 are actual lines from Example 2's error.log, one from each of three functions.

The final SIGIO line needed explaining while writing this post. The master sets FIOASYNC and F_SETOWN (owner = master) on its end of the channel (channel[0]) (lines 146–158). When the worker ended and the other end closed, the kernel apparently treated that socket as "readable" (EOF) and sent SIGIO to the master. That is kernel behavior, not something confirmed in the nginx source. In this build the master only sets ngx_sigio = 1 on SIGIO, and the only code that uses it is inside #if (NGX_CONTROL_API) (lines 177–196), so nothing happens.

Easy to get wrong: one SIGCHLD can reap two children

If the same signal arrives again while it is still pending, the two merge into one. In section 4's example (the run shown in this post), a single signal 17 (SIGCHLD) received from 40 line was followed by both worker process 39 exited and worker process 40 exited. That is why ngx_process_get_status() doesn't call waitpid() once but repeats it with WNOHANG until it returns 0. Seeing only "from 40" in the log, you don't need to go looking for where 39 went. Whether they merge depends on when the two children exit; when I ran the same example again, they were reaped by two separate SIGCHLD lines.

4. Zero-downtime reload — start the new workers first, let the old ones finish

The master side: ngx_reconfigure

When ngx_reconfigure is set, the master loop does this:

        if (ngx_reconfigure) {
            ngx_reconfigure = 0;

            if (ngx_new_binary) {
                ngx_start_worker_processes(cycle, ccf->worker_processes,
                                           NGX_PROCESS_RESPAWN);
                ngx_start_cache_manager_processes(cycle, 0);
                ngx_noaccepting = 0;

                continue;
            }

            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reconfiguring");

            cycle = ngx_init_cycle(cycle);
            if (cycle == NULL) {
                cycle = (ngx_cycle_t *) ngx_cycle;
                continue;
            }

            ngx_cycle = cycle;
            ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
                                                   ngx_core_module);
            ngx_start_worker_processes(cycle, ccf->worker_processes,
                                       NGX_PROCESS_JUST_RESPAWN);
            ngx_start_cache_manager_processes(cycle, 1);

            /* allow new processes to start */
            ngx_msleep(100);

            live = 1;
            ngx_signal_worker_processes(cycle,
                                        ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
        }

src/os/unix/ngx_process_cycle.c lines 233–266

Read in order, this is all of reload (the ngx_new_binary branch is section 5).

1. ngx_init_cycle(cycle) builds a new cycle from the old one. It is the same function from Part 4 that reads the config from scratch. One reason the old cycle is passed in is the listening sockets — if a listen in the new config has the same address as an old one, the old fd is taken over instead of opening a new socket (nls[n].fd = ls[i].fd;, src/core/ngx_cycle.c lines 535–541). Because the socket is never closed and reopened, connections that land in the kernel's queue during the reload are not lost.
2. On failure, nothing changes. If it returns NULL, the loop goes back to the global ngx_cycle (the old cycle) and continues. The old workers keep working.
3. New workers are started with NGX_PROCESS_JUST_RESPAWN, i.e. respawn = 1, just_spawn = 1. A new worker inherits the memory as of fork() — the new cycle. If you changed worker_processes, the new count starts here.
4. It sleeps 100 milliseconds. As the comment says, "allow new processes to start".
5. ngx_signal_worker_processes(SHUTDOWN) sends QUIT to every child but skips the ones just started. We look at this next.

QUIT goes over the channel, not as a signal

ngx_signal_worker_processes() first turns the signal number into a channel command: SIGQUIT → NGX_CMD_QUIT, SIGTERM → NGX_CMD_TERMINATE, SIGUSR1 → NGX_CMD_REOPEN, anything else (SIGKILL etc.) → 0 (src/os/unix/ngx_process_cycle.c lines 468–484). Then for each child:

        if (ngx_processes[i].detached || ngx_processes[i].pid == -1) {
            continue;
        }

        if (ngx_processes[i].just_spawn) {
            ngx_processes[i].just_spawn = 0;
            continue;
        }

        if (ngx_processes[i].exiting
            && signo == ngx_signal_value(NGX_SHUTDOWN_SIGNAL))
        {
            continue;
        }

        if (ch.command) {
            if (ngx_write_channel(ngx_processes[i].channel[0],
                                  &ch, sizeof(ngx_channel_t), cycle->log)
                == NGX_OK)
            {
                if (signo != ngx_signal_value(NGX_REOPEN_SIGNAL)) {
                    ngx_processes[i].exiting = 1;
                }

                continue;
            }
        }

src/os/unix/ngx_process_cycle.c lines 503–529

• A just_spawn slot is skipped this one time and the bit is cleared. This single bit makes "keep the new workers, send away only the old ones" work. Since the bit is cleared, at the next reload this worker becomes an old worker and gets QUIT.
• QUIT, TERMINATE and REOPEN go over the channel first. If the channel write succeeds, kill() is not called (the kill() further down at line 534 is reached only when the channel write failed, or when there is no channel command, as with SIGKILL). After sending, exiting = 1 is recorded so ngx_reap_children() won't revive that worker when it ends.

The 32 bytes sent over the channel are this struct:

typedef struct {
    ngx_uint_t  command;
    ngx_pid_t   pid;
    ngx_int_t   slot;
    ngx_fd_t    fd;
} ngx_channel_t;

src/os/unix/ngx_channel.h lines 17–22

The command numbers are NGX_CMD_OPEN_CHANNEL 1, CLOSE_CHANNEL 2, QUIT 3, TERMINATE 4, REOPEN 5 (src/os/unix/ngx_process_cycle.h lines 16–20). On 64-bit Linux ngx_uint_t and ngx_int_t are 8 bytes, and pid and fd are 4 bytes each plus 4 bytes of alignment padding, for 32 bytes total. On the worker side the channel fd is registered in the event loop like any other connection (ngx_add_channel_event(), lines 972–978), and when there is something to read, ngx_channel_handler() is called and sets ngx_quit = 1 for NGX_CMD_QUIT, ngx_terminate = 1 for TERMINATE, and ngx_reopen = 1 for REOPEN (src/os/unix/ngx_process_cycle.c lines 1089–1101).

These are the same flags the signal handler sets. So from the worker's point of view, whether QUIT came by kill -QUIT or over the channel, the path afterwards is the same. The source doesn't say why the channel is tried first. What the code does show is the difference: unlike signals, channel messages don't merge but queue up in order, and they are handled inside the event loop like any other event.

The worker side: gracefully shutting down

    for ( ;; ) {

        if (ngx_exiting) {
            if (ngx_event_no_timers_left() == NGX_OK) {
                ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
                ngx_worker_process_exit(cycle);
            }
        }

        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "worker cycle");

        ngx_process_events_and_timers(cycle);

        if (ngx_terminate) {
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
            ngx_worker_process_exit(cycle);
        }

        if (ngx_quit) {
            ngx_quit = 0;
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0,
                          "gracefully shutting down");
            ngx_setproctitle("worker process is shutting down");

            if (!ngx_exiting) {
                ngx_exiting = 1;
                ngx_set_shutdown_timer(cycle);
                ngx_close_listening_sockets(cycle);
                ngx_close_idle_connections(cycle);
                ngx_event_process_posted(cycle, &ngx_posted_events);
            }
        }

src/os/unix/ngx_process_cycle.c lines 740–771

A worker that sees ngx_quit does the following, once only (!ngx_exiting):

1. ngx_exiting = 1 — from now on it is "leaving". HTTP modules look at this global too. When building response headers, if r->keepalive && (ngx_terminate || ngx_exiting), keep-alive is turned off (src/http/ngx_http_header_filter_module.c lines 204–206), which is why responses in progress get Connection: close. The place where a finished request moves into keep-alive also checks !ngx_exiting (src/http/ngx_http_request.c lines 3054–3061).
2. ngx_close_listening_sockets() closes this worker's listening sockets. The master and the new workers still hold the same sockets, so the port doesn't close; only this worker stops receiving new connections.
3. ngx_close_idle_connections() immediately closes connections that are idle, waiting for a request (c->idle) (src/core/ngx_connection.c lines 1461–1478).

The way it closes them is very nginx. It walks the connection array and, instead of calling close() directly, sets c->close = 1 and calls that connection's read handler. For an HTTP keep-alive connection, ngx_http_keepalive_handler() is called, sees c->close, and cleans up by itself. Each connection is left to whoever knows how to tear it down (the module). For HTTP/1.x connections, idle is turned on in ngx_http_set_keepalive() (src/http/ngx_http_request.c lines 3556–3559). If you set keepalive_min_timeout (default 0), though, it is turned on only after that time has passed, in ngx_http_keepalive_handler() (lines 3593–3599), so keep-alive connections younger than that are not closed here.

What remains are the connections in progress. The worker just keeps looping and handling them, and leaves when, at the top of the loop, ngx_event_no_timers_left() returns NGX_OK.

    /* ... NGX_OK if the tree is empty ... */

    for (node = ngx_rbtree_min(root, sentinel);
         node;
         node = ngx_rbtree_next(&ngx_event_timer_rbtree, node))
    {
        ev = ngx_rbtree_data(node, ngx_event_t, timer);

        if (!ev->cancelable) {
            return NGX_AGAIN;
        }
    }

    /* only cancelable timers left */

    return NGX_OK;
}

src/event/ngx_event_timer.c lines 112–126 (the whole function is lines 99–126)

"Are there connections in progress?" is decided by timers, not by counting connections. nginx's timers all live in one red-black tree (a balanced binary tree where insert, delete and find-minimum are all fast) (Part 3; details in Part 7). A connection in progress usually has a read or write timeout timer (proxy_read_timeout, send_timeout and the like), so "no non-cancelable timer left" means "nothing left to wait for". Timers that need no waiting, such as the one that flushes log buffers or the resolver's timers, are marked cancelable = 1 and so don't count.

A graceful shutdown waits not for "zero connections" but for "zero timers to wait for".

If there are connections that never end (a very long download, WebSocket and so on), the old worker stays around a long time. That is what worker_shutdown_timeout is for; its default is 0 (off) (ngx_conf_init_msec_value(ccf->shutdown_timeout, 0), src/core/nginx.c line 1194). If you set it, ngx_set_shutdown_timer() arms a timer for that long, and when it fires ngx_shutdown_timer_handler() sets close and error on each remaining connection and calls its read handler to finish it (src/core/ngx_cycle.c lines 1436–1484). That timer itself is cancelable = 1, so it doesn't hold the worker back from leaving.

Putting the whole sequence on one page:

Part 2's reload picture, zoomed in to the function level.

Example 3: reload with an idle connection and a request in progress

Continuing ex/u06-reload from section 1: after start, run.sh (1) opens one keep-alive connection, sends one request and leaves it idle (idle.py), (2) sends a /slow request to a backend that answers 3 seconds late (slow.py), and while that is in flight, (3) changes v1 to v2 in the config and runs nginx -s reload. Times are seconds since start.

Here is the output. The first part shown in section 1, error.log's startup lines and the final -s quit part are cut to ....

=== 2. an idle keep-alive connection and an in-flight request (/slow, 3s) ===
 0.73s idle conn: got 'v1 pid=20\n'
=== 3. edit v1 -> v2 and nginx -s reload ===
 1.46s reload
 1.59s idle conn: closed by server
    PID    PPID USER     CMD
     19       1 root     nginx: master process nginx -c /tmp/u06/nginx.conf
     20      19 nobody   nginx: worker process is shutting down
     39      19 nobody   nginx: worker process
     40      19 nobody   nginx: worker process
 2.06s curl /hello: v2 pid=39
 3.94s /slow finished
--- response of /slow (headers)
HTTP/1.1 200 OK
Server: nginx/1.31.9
Date: Sat, 07 Nov 2026 02:51:16 GMT
Content-Length: 14
Connection: close

slow response
=== 4. after the old worker finished ===
    PID    PPID USER     CMD
     19       1 root     nginx: master process nginx -c /tmp/u06/nginx.conf
     39      19 nobody   nginx: worker process
     40      19 nobody   nginx: worker process
=== error.log ===
...
2026/11/07 02:51:14 [notice] 19#19: signal 1 (SIGHUP) received from 38, reconfiguring
2026/11/07 02:51:14 [notice] 19#19: reconfiguring
2026/11/07 02:51:14 [notice] 19#19: using the "epoll" event method
2026/11/07 02:51:14 [notice] 19#19: start worker processes
2026/11/07 02:51:14 [notice] 19#19: start worker process 39
2026/11/07 02:51:14 [notice] 19#19: start worker process 40
2026/11/07 02:51:14 [notice] 21#21: gracefully shutting down
2026/11/07 02:51:14 [notice] 21#21: exiting
2026/11/07 02:51:14 [notice] 21#21: exit
2026/11/07 02:51:14 [notice] 20#20: gracefully shutting down
2026/11/07 02:51:14 [notice] 19#19: signal 17 (SIGCHLD) received from 21
2026/11/07 02:51:14 [notice] 19#19: worker process 21 exited with code 0
2026/11/07 02:51:14 [notice] 19#19: signal 29 (SIGIO) received
2026/11/07 02:51:16 [notice] 20#20: exiting
2026/11/07 02:51:16 [notice] 20#20: exit
2026/11/07 02:51:16 [notice] 19#19: signal 17 (SIGCHLD) received from 20
2026/11/07 02:51:16 [notice] 19#19: worker process 20 exited with code 0
2026/11/07 02:51:16 [notice] 19#19: signal 29 (SIGIO) received
2026/11/07 02:51:17 [notice] 19#19: signal 3 (SIGQUIT) received from 53, shutting down
...
2026/11/07 02:51:17 [notice] 19#19: signal 17 (SIGCHLD) received from 40
2026/11/07 02:51:17 [notice] 19#19: worker process 39 exited with code 0
2026/11/07 02:51:17 [notice] 19#19: worker process 40 exited with code 0
2026/11/07 02:51:17 [notice] 19#19: exit

pids, times and which worker gets which connection differ on every run. In this run both the idle connection and the /slow request went to worker 20.

1. The idle keep-alive connection was closed 0.13 s after the reload (1.46s → 1.59s), without waiting for keepalive_timeout 60s. That is ngx_close_idle_connections(). To the client it looks like "the server closed the connection", so a client reusing keep-alive connections has to send its next request on a new connection (HTTP client libraries usually retry this).
2. Worker 20 stayed as worker process is shutting down, because it had the /slow request in progress. Worker 21, which had nothing to do, went through gracefully shutting down → exiting → exit within the same second.
3. The new request was handled by new worker 39 with the new config (v2). The old worker closed its listening socket, so new connections don't go to it.
4. /slow completed without interruption, and the response got Connection: close. The header filter saw ngx_exiting and turned keep-alive off. After sending the response, worker 20 had no timers left, logged exiting and ended (02:51:16).
5. The workers' logs have no signal 3 (SIGQUIT) received — neither at reload nor at the final -s quit. Only the master received signals; the workers got QUIT over the channel. We confirm it with strace in the next example.
6. At the final shutdown, one SIGCHLD line reaped two workers (section 3's "Easy to get wrong"). This depends on timing; in another run they came as two lines.

Example 4: system calls of the master and an old worker during reload

The third part of ex/u06-respawn attaches strace to the master and one old worker during a reload. For the master we traced only rt_sigsuspend,socketpair,clone,sendmsg,kill,wait4, for the worker only epoll_wait,recvmsg,close,exit_group. The lines are long, so the tail of each is cut to ….

--- master 11
rt_sigsuspend([], 8)                    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=48, si_uid=0} ---
socketpair(AF_UNIX, SOCK_STREAM, 0, [4, 10]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0212a5da10) = 50
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0", …
sendmsg(7, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0", …
socketpair(AF_UNIX, SOCK_STREAM, 0, [11, 12]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0212a5da10) = 51
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0003\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0", …
sendmsg(7, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0003\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0", …
sendmsg(4, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0003\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0", …
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0", …
sendmsg(7, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0", …
rt_sigsuspend([], 8)                    = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=22, si_uid=65534, si_status=0, si_utime=0, si_stime=0} ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 22
wait4(-1, 0x7ffe6ffe9dbc, WNOHANG, NULL) = 0
sendmsg(7, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\2\0\0\0\0\0\0\0\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0", …
sendmsg(4, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\2\0\0\0\0\0\0\0\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0", …
sendmsg(11, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\2\0\0\0\0\0\0\0\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0", …
rt_sigsuspend([], 8)                    = ? ERESTARTNOHAND (To be restarted if no handler)
...
--- old worker 13
epoll_wait(9, [{events=EPOLLIN, data={u32=509870544, u64=94584279663056}}], 512, -1) = 1
recvmsg(8, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0", …
recvmsg(8, {msg_namelen=0}, 0)          = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(9, [{events=EPOLLIN, data={u32=509870544, u64=94584279663056}}], 512, -1) = 1
recvmsg(8, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\0\0\0\0\0\0003\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0", …
recvmsg(8, {msg_namelen=0}, 0)          = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(9, [{events=EPOLLIN, data={u32=509870544, u64=94584279663056}}], 512, -1) = 1
recvmsg(8, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0", …
recvmsg(8, {msg_namelen=0}, 0)          = -1 EAGAIN (Resource temporarily unavailable)
close(5)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

fd numbers, pids and addresses differ on every run. Reading the 32 bytes of iov_base as ngx_channel_t (strace prints bytes as octal escapes: \1=1, \v=11, \26=22 (octal 26). \0002 is \000 followed by the character 2 (0x32 = 50)):

1. The master's strace has no kill() at all. QUIT went as two sendmsg() calls (two old workers), and the new workers (fds 4 and 11) got no QUIT — that's just_spawn.
2. There is one socketpair() → clone() pair per worker. glibc's fork() shows up as the clone() system call on Linux. It is exactly the order in ngx_spawn_process() from section 1.
3. The old worker got a read event on the channel (fd 8) from epoll_wait() and read it with recvmsg(). There is no --- SIGQUIT --- line. The third message is QUIT (3), immediately followed by close(5) and then exit_group(0). fd 5 is the listening socket (we started the same config again and checked the worker's /proc/<pid>/fd and /proc/net/tcp) — that's ngx_close_listening_sockets(). With no connections in progress, there were no timers to wait for. You can also see that right up to leaving, this worker received the new workers' OPEN_CHANNEL messages.

The master's debug log for the same reload shows the ledger bits (only some of the master pid's lines are picked):

2026/11/07 02:51:06 [notice] 11#11: signal 1 (SIGHUP) received from 48, reconfiguring
2026/11/07 02:51:06 [debug] 11#11: wake up, sigio 0
2026/11/07 02:51:06 [notice] 11#11: reconfiguring
2026/11/07 02:51:06 [notice] 11#11: start worker processes
2026/11/07 02:51:06 [notice] 11#11: start worker process 50
2026/11/07 02:51:06 [debug] 11#11: pass channel s:2 pid:50 fd:4 to s:0 pid:22 fd:3
2026/11/07 02:51:06 [debug] 11#11: pass channel s:2 pid:50 fd:4 to s:1 pid:13 fd:7
2026/11/07 02:51:06 [notice] 11#11: start worker process 51
2026/11/07 02:51:06 [debug] 11#11: pass channel s:3 pid:51 fd:11 to s:0 pid:22 fd:3
2026/11/07 02:51:06 [debug] 11#11: pass channel s:3 pid:51 fd:11 to s:1 pid:13 fd:7
2026/11/07 02:51:06 [debug] 11#11: pass channel s:3 pid:51 fd:11 to s:2 pid:50 fd:4
2026/11/07 02:51:06 [debug] 11#11: child: 0 22 e:0 t:0 d:0 r:1 j:0
2026/11/07 02:51:06 [debug] 11#11: child: 1 13 e:0 t:0 d:0 r:1 j:0
2026/11/07 02:51:06 [debug] 11#11: child: 2 50 e:0 t:0 d:0 r:1 j:1
2026/11/07 02:51:06 [debug] 11#11: child: 3 51 e:0 t:0 d:0 r:1 j:1
2026/11/07 02:51:06 [debug] 11#11: sigsuspend
2026/11/07 02:51:07 [notice] 11#11: signal 17 (SIGCHLD) received from 22
2026/11/07 02:51:07 [notice] 11#11: worker process 22 exited with code 0
2026/11/07 02:51:07 [debug] 11#11: wake up, sigio 0
2026/11/07 02:51:07 [debug] 11#11: reap children
2026/11/07 02:51:07 [debug] 11#11: child: 0 22 e:1 t:1 d:0 r:1 j:0
2026/11/07 02:51:07 [debug] 11#11: child: 1 13 e:1 t:0 d:0 r:1 j:0
...

The format is child: slot pid e:exiting t:exited d:detached r:respawn j:just_spawn (lines 493–501 of ngx_signal_worker_processes(); lines 571–579 of ngx_reap_children() print the same format). Just before QUIT was sent, new workers 50 and 51 were j:1; after it, old workers 22 and 13 became e:1 and the new workers went back to j:0. The finished 22 is t:1, but since it is e:1, ngx_reap_children() didn't revive it. Worker 22 is the one revived after kill -9 in example 2.

Easy to get wrong: reloading with a broken config

The second part of ex/u06-respawn appends oops; to the config and reloads.

=== 2. reload with a broken config ===
--- nginx -s reload
nginx: [emerg] unknown directive "oops" in /tmp/u06r/nginx.conf:22
(exit code 1)
--- kill -HUP 11
    PID    PPID USER     CMD
     11       1 root     nginx: master process nginx -c /tmp/u06r/nginx.conf
     13      11 nobody   nginx: worker process
     22      11 nobody   nginx: worker process
--- error.log (non-debug lines)
2026/11/07 02:51:04 [notice] 11#11: signal 1 (SIGHUP) received from 1, reconfiguring
2026/11/07 02:51:04 [notice] 11#11: reconfiguring
2026/11/07 02:51:04 [emerg] 11#11: unknown directive "oops" in /tmp/u06r/nginx.conf:22

1. nginx -s reload never even sent the signal. A process started with -s also reads the config first with ngx_init_cycle() in main() (because the pid file path is in the config), and if that fails it ends with return 1. Sending the signal (ngx_signal_process()) comes after that (src/core/nginx.c lines 300–338).
2. Even sending kill -HUP straight to the master leaves the server as it was. The master logs reconfiguring, fails in ngx_init_cycle(), and stays on the old cycle through the cycle == NULL branch in section 4's code. The pids of workers 13 and 22 didn't change.

httpd, as seen in httpd Part 5, differs here. If you send kill -USR1 straight to the parent for a graceful restart and the new config has an error, httpd handles every failure inside the main() loop with destroy_and_exit_process(), so the running parent exits. nginx builds the new cycle separately next to the old one and swaps only once it is complete, so a failure costs nothing.

Easy to get wrong: who frees the old cycle?

Each reload creates a new cycle. The master frees the old cycle's pool right away at the end of ngx_init_cycle().

    if (ngx_process == NGX_PROCESS_MASTER || ngx_is_init_cycle(old_cycle)) {

        ngx_destroy_pool(old_cycle->pool);
        cycle->old_cycle = NULL;

        return cycle;
    }

src/core/ngx_cycle.c lines 783–789

Is that safe while old workers are still handling requests with the old config? Yes — an old worker uses the old cycle in its own memory, copied at fork(). The master freeing its copy doesn't affect the old worker's copy. The master holds no connections, so it has no reason to keep the old cycle. The ngx_old_cycles below it (lines 792–828) and ngx_clean_old_cycles(), which runs every 30 seconds, are for reloading in single-process mode (master_process off), where there is no master. There, the old cycle's connections stay inside the same process, so it waits until they are all closed before freeing it.

5. Binary upgrade — USR2, WINCH, QUIT

Reload changes only the config with the same nginx binary. Upgrading nginx itself means running a new program, and closing and reopening the listening sockets would lose the connections in between. nginx hands the open sockets to the new binary.

The old master is kept so you can roll back. If something is wrong, send HUP to the old master to start the old workers again and take the new master down (the ngx_new_binary branch in section 4's code).

ngx_exec_new_binary()

When the master loop sees ngx_change_binary, it calls ngx_exec_new_binary(cycle, ngx_argv) (lines 284–288). Here is the part that hands over the listening sockets:

    p = ngx_cpymem(var, NGINX_VAR "=", sizeof(NGINX_VAR));

    ls = cycle->listening.elts;
    for (i = 0; i < cycle->listening.nelts; i++) {
        if (ls[i].ignore) {
            continue;
        }
        p = ngx_sprintf(p, "%ud;", ls[i].fd);
    }

src/core/nginx.c lines 750–758

NGINX_VAR is "NGINX" (src/core/nginx.h line 22). So the listening sockets' fd numbers are joined with ; into an environment variable like NGINX=5;. fds survive execve() (no FD_CLOEXEC is set on listening sockets), so the new program can use the same sockets just by knowing the numbers. Next it renames the pid file to nginx.pid.oldbin (line 799) and does fork() + execve(argv[0], ...) through ngx_execute() (line 811). ngx_execute() is ngx_spawn_process(..., NGX_PROCESS_DETACHED), so the new binary is recorded as detached and has no channel.

The new binary's main() reads getenv("NGINX") in ngx_add_inherited_sockets(), puts those fds into cycle->listening with inherited = 1, and sets ngx_inherited = 1 (src/core/nginx.c lines 481–538). With ngx_inherited set it skips the daemonizing fork() (lines 356–366), so the new master stays a child of the old master.

Example 5: swap the binary with USR2

ex/u06-binary-upgrade uses the same binary as the "new binary" and walks through the sequence. First we start it as nginx found through PATH, then with the full path.

=== 1. started as 'nginx' (found through PATH), then USR2 ===
    PID    PPID CMD
     10       1 nginx: master process nginx -c /work/nginx.conf
     11      10 nginx: worker process
/tmp/u06b/nginx.pid: 10
2026/11/07 02:53:38 [notice] 10#10: signal 12 (SIGUSR2) received from 1, changing binary
2026/11/07 02:53:38 [notice] 10#10: changing binary
2026/11/07 02:53:38 [notice] 10#10: start new binary process 16
2026/11/07 02:53:38 [alert] 16#16: execve() failed while executing new binary process "nginx" (2: No such file or directory)
2026/11/07 02:53:38 [notice] 10#10: new binary process 16 exited with code 1
=== 2. started as /opt/nginx/sbin/nginx, then USR2 ===
--- kill -USR2 23
    PID    PPID CMD
     23       1 nginx: master process /opt/nginx/sbin/nginx -c /work/nginx.conf
     24      23 nginx: worker process
     29      23 nginx: master process /opt/nginx/sbin/nginx -c /work/nginx.conf
     30      29 nginx: worker process
/tmp/u06b/nginx.pid: 29
/tmp/u06b/nginx.pid.oldbin: 23
curl /hello: pid=24
curl /hello: pid=24
curl /hello: pid=30
curl /hello: pid=30
--- kill -WINCH 23  (old workers stop, old master stays)
    PID    PPID CMD
     23       1 nginx: master process /opt/nginx/sbin/nginx -c /work/nginx.conf
     29      23 nginx: master process /opt/nginx/sbin/nginx -c /work/nginx.conf
     30      29 nginx: worker process
curl /hello: pid=30
--- kill -QUIT 23  (old master exits)
    PID    PPID CMD
     29       1 nginx: master process /opt/nginx/sbin/nginx -c /work/nginx.conf
     30      29 nginx: worker process
/tmp/u06b/nginx.pid: 29
...
=== the default error log (/opt/nginx/logs/error.log) ===
2026/11/07 02:53:40 [notice] 29#29: using inherited sockets from "5;"

pids and times differ on every run. The full error.log in the middle is cut to ....

1. Started under the name nginx, USR2 fails. Unlike a shell, execve() doesn't search PATH; it looks for "nginx", the value of ctx.path = argv[0] (line 732), as a path relative to the current directory. The master's current directory (/tmp) has no nginx, hence No such file or directory. The failed new-binary process ends with exit(1), and the master, in ngx_reap_children(), recognizes it as ngx_new_binary and renames nginx.pid.oldbin back to nginx.pid (lines 639–652). That's why there is only one pid file, nginx.pid: 10. If you plan to use binary upgrades, start nginx with its full path.
2. Started with the full path, there are two masters. The new master 29's parent is the old master 23, and there are two pid files, nginx.pid (29) and nginx.pid.oldbin (23). Requests were split between old worker 24 and new worker 30 (which one gets how many varies from run to run) — two generations of workers accept()ing on the same socket.
3. On WINCH, the old master sends its workers away but stays. ngx_noaccept → SHUTDOWN to the workers (lines 290–295). It remembers ngx_noaccepting = 1, and if the new binary fails and ends, it starts the old workers again through ngx_restart (lines 659–662). It leaves a way back.
4. When the old master exits on QUIT, the new master's PPID becomes 1. A process whose parent ends is adopted by another process (here, process 1 in the container).
5. using inherited sockets from "5;" was written not to the config's error.log but to the default log /opt/nginx/logs/error.log. ngx_add_inherited_sockets() runs before the config is read, so the log at that point is the default path set at configure time. This is where we got stuck — at first we only looked at the config's error.log, didn't find this line, and thought the sockets hadn't been inherited.

Easy to get wrong: sending USR2 to the new master again is ignored

        case ngx_signal_value(NGX_CHANGEBIN_SIGNAL):
            if (ngx_getppid() == ngx_parent || ngx_new_binary > 0) {

                /* ... comment: ignore the signal in the new binary while the old one
                   still runs, or in the old binary once the new one runs ... */

                action = ", ignoring";
                ignore = 1;
                break;
            }

src/os/unix/ngx_process.c lines 374–387

The new master records ngx_parent = ngx_getppid() (the old master) in main() (src/core/nginx.c line 236). If its parent is still the same, the old master is alive, so USR2 is ignored; and the old master ignores it too once it has already started a new binary (ngx_new_binary > 0). This prevents three generations from overlapping. For an ordinary daemonized master, ngx_daemon() sets ngx_parent to its own pre-daemon pid (ngx_daemon.c line 29), which differs from the real parent, so USR2 is accepted.

6. A worker's first moments — ngx_worker_process_init()

This is what a worker does right after fork(), before entering the event loop. The part that drops privileges:

    if (geteuid() == 0) {
        if (setgid(ccf->group) == -1) {
            ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
                          "setgid(%d) failed", ccf->group);
            /* fatal */
            exit(2);
        }

        if (initgroups(ccf->username, ccf->group) == -1) {

            if (ngx_errno != NGX_EPERM) {
                /* ... "initgroups(%s, %d) failed" ... */
                exit(2);
            }

            /* ... "initgroups(%s, %d) failed, ignored" ... */
        }

        /* ... PR_SET_KEEPCAPS for transparent proxying ... */

        if (setuid(ccf->user) == -1) {
            ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
                          "setuid(%d) failed", ccf->user);
            /* fatal */
            exit(2);
        }

src/os/unix/ngx_process_cycle.c lines 833–872

The order setgid() → initgroups() → setuid() cannot be changed: after setuid() drops root, there is no permission left to change groups. Every failure is exit(2) — the "don't restart" exit code from section 3. The base commit of this post is exactly the commit that made initgroups() failures fatal ("Core: made initgroups() failures fatal"). Now only EPERM (not permitted) logs an alert-level message ("…failed, ignored") and continues; any other failure ends the worker.

The three steps of dropping privileges and the path taken on failure. The red boxes end in the "exit code 2" handling from section 3.

The whole of ngx_worker_process_init() goes in this order (lines 804–978):

Step 6 matters. A worker's epoll instance and its array of worker_connections connections are created by each worker separately in init_process, not by the master (ngx_event_process_init(), Part 7). Workers share only the listening sockets and shared memory zones; connections and events are per worker.

What this post leaves out

• Single-process mode ngx_single_process_cycle() (master_process off): src/os/unix/ngx_process_cycle.c lines 300–354
• The cache manager and cache loader processes (helper processes started when there is a proxy_cache_path): ngx_start_cache_manager_processes() lines 374–414, ngx_cache_manager_process_cycle() from line 1138
• The control API (--with-control-api; nginx -l <address> exposes information about the running processes over HTTP): it came into 1.31.x with commit 4bc459a15 "Control API." and is not in 1.30.6. It is disabled in this build, so the #if (NGX_CONTROL_API) blocks (lines 177–197 and others) are not compiled. The code is in src/os/unix/ngx_control.c
• Cleanup in ngx_master_process_exit() and ngx_worker_process_exit() (module exit_master / exit_process, and ngx_exit_cycle, which keeps logging possible after exit): lines 681–725, 982–1041
• Log rotation, ngx_reopen_files(): src/core/ngx_cycle.c
• Stopping or leaving a core dump on abnormal exit with debug_points (ngx_debug_point()): src/os/unix/ngx_process.c lines 611–628
• Master and workers on Windows (event objects instead of signals): src/os/win32/ngx_process_cycle.c

Summary

What's next

We've followed the worker up to the point where it enters ngx_worker_process_cycle(). Inside one line of that loop, ngx_process_events_and_timers(cycle), epoll watches thousands of connections, the red-black tree timers are processed, and new connections are accept()ed. The timer tree we only called "timers to wait for" in this post, and the very event loop the channel was plugged into, are the stars of Part 7: The Event Loop — epoll, Timers, Posted Events and accept.

Part 8: MPM of the httpd series does the same job — a parent starting, counting and sending away children — differently. httpd's parent checks the scoreboard (a shared-memory status board) every second to grow and shrink the number of children, and to send a child away it writes one byte into a pipe called the POD (pipe of death). nginx's master keeps a fixed number of workers and wakes only when a signal arrives, and to send workers away it writes a command into each worker's own channel. The idea of saying "stop" through a pipe is similar; the difference is that httpd leaves who stops to whichever child reads first, while nginx's master picks each one. The httpd side of re-reading the config on a graceful restart is in httpd Part 5: Startup and main().

• Part 2: The Big Picture — Master and Workers, Modules, the Cycle and Config Contexts
• Part 4: Startup and Reading the Config — From main() to ngx_init_cycle
• Tomcat Part 5: The Connector and the NIO Endpoint — a Java server's answer, splitting the same work among threads inside one process