Shedding old code with ecdysis: graceful restarts for Rust services at Cloudflare

The upgrade problem at network scale

Cloudflare's Rust services handle millions of requests per second across hundreds of data centers. When those services need a security patch or a new feature, the team can't afford even a brief service interruption. A naive restart — stopping the old process, starting the new one — closes listening sockets, refuses new connections with ECONNREFUSED, and terminates every established connection mid-stream. For long-lived connections like WebSockets or gRPC streams, that's an abrupt cutoff from the client's perspective.

For a service processing thousands of requests per second at a single location, a sub-second gap can mean hundreds of dropped connections. Multiply that across Cloudflare's global footprint and a routine restart becomes millions of failed requests.

The obvious workaround — starting the new process before stopping the old one — runs into a different problem. The SO_REUSEPORT socket option allows multiple processes to bind to the same address and port, but the kernel load-balances incoming SYN packets across the sockets. When a process exits before accepting the connections assigned to it, those connections become orphaned and are killed. GitHub documented the same issue while building its GLB Director load balancer.

After five years of production use, Cloudflare has open-sourced ecdysis, a Rust library implementing graceful process restarts that avoid both failure modes: no existing connection is dropped, no new connection is refused.

The fork-and-exec approach

ecdysis follows a model NGINX has used for years. The upgrade sequence is:

1. The parent process fork()s a new child.
2. The child replaces itself with new code via execve().
3. The child inherits socket file descriptors through a named pipe.
4. The parent waits for a readiness signal before shutting down.

Because the listening socket stays open throughout the transition, both processes share the same underlying kernel data structure during the child's initialization. The parent keeps accepting and handling new and existing connections until the child signals readiness, at which point the parent closes its copy of the listening socket and continues processing only existing connections. Any connection accepted by the parent during the overlap is simply completed as part of the drain.

This design satisfies four design goals Cloudflare set when building the library: old code fully shuts down after an upgrade, the new process gets an initialization grace period, a crash in new code during initialization doesn't affect the running service, and only a single upgrade runs at a time. If the child fails during setup — say, due to a configuration error — it simply exits, and the parent never stopped listening.

Because ecdysis relies on Unix-specific syscalls for socket inheritance and process management, it does not work on Windows.

Integration with Tokio and systemd

ecdysis is built as a first-class citizen of the async Rust ecosystem:

• Tokio integration: Inherited sockets become async stream wrappers usable as listeners without extra glue code. Synchronous services can run without any async runtime.
• systemd-notify support: With the systemd_notify feature enabled, ecdysis integrates with systemd lifecycle notifications. Setting Type=notify-reload in the service unit file lets systemd track upgrades.
• systemd named sockets: The systemd_sockets feature manages systemd-activated sockets, allowing a service to be socket-activated and support graceful restarts at the same time.

Security notes

The fork model briefly runs two process generations with access to the same listening sockets. ecdysis mitigates concerns through several design choices:

• Fork-then-exec: The child starts with a fresh address space and new code, inheriting no memory from the parent — only explicitly passed file descriptors.
• Explicit inheritance: Only listening sockets and communication pipes cross the boundary; other descriptors are closed via CLOEXEC.
• seccomp compatibility: Services using seccomp filters must allow fork() and execve().

These tradeoffs are well understood; the fork-exec pattern is battle-tested in long-running software like NGINX and Apache.

A minimal example

Here is a simplified TCP echo server with graceful restart support:

use ecdysis::tokio_ecdysis::{SignalKind, StopOnShutdown, TokioEcdysisBuilder};
use tokio::{net::TcpStream, task::JoinSet};
use futures::StreamExt;
use std::net::SocketAddr;

#[tokio::main]
async fn main() {
    // Create the ecdysis builder
    let mut ecdysis_builder = TokioEcdysisBuilder::new(
        SignalKind::hangup()  // Trigger upgrade/reload on SIGHUP
    ).unwrap();

    // Trigger stop on SIGUSR1
    ecdysis_builder
        .stop_on_signal(SignalKind::user_defined1())
        .unwrap();

    // Create listening socket - will be inherited by children
    let addr: SocketAddr = "0.0.0.0:8080".parse().unwrap();
    let stream = ecdysis_builder
        .build_listen_tcp(StopOnShutdown::Yes, addr, |builder, addr| {
            builder.set_reuse_address(true)?;
            builder.bind(&addr.into())?;
            builder.listen(128)?;
            Ok(builder.into())
        })
        .unwrap();

    // Spawn task to handle connections
    let server_handle = tokio::spawn(async move {
        let mut stream = stream;
        let mut set = JoinSet::new();
        while let Some(Ok(socket)) = stream.next().await {
            set.spawn(handle_connection(socket));
        }
        set.join_all().await;
    });

    // Signal readiness and wait for shutdown
    let (_ecdysis, shutdown_fut) = ecdysis_builder.ready().unwrap();
    let shutdown_reason = shutdown_fut.await;

    log::info!("Shutting down: {:?}", shutdown_reason);

    // Gracefully drain connections
    server_handle.await.unwrap();
}

async fn handle_connection(mut socket: TcpStream) {
    // Echo connection logic here
}

Three calls matter most. build_listen_tcp creates a listener the child process will inherit. ready() signals to the parent that initialization is complete and it can exit safely. shutdown_fut.await blocks until an upgrade or stop is requested.

On receiving SIGHUP, the parent forks and execs a new instance, passes the listening socket, and waits for the child's ready() call before draining and exiting. The child goes through the same initialization flow, except inherited ecdysis sockets aren't rebound; it signals readiness, then blocks on a shutdown or upgrade signal.

Production history and alternatives

ecdysis has run in production at Cloudflare since 2021, deployed across more than 330 data centers in 120-plus countries. Services using it handle billions of requests per day, with updates deployed frequently for security patches and features. Every restart preserves hundreds of thousands of requests that would otherwise be dropped in a stop/start cycle.

Cloudflare's Go library tableflip uses the same fork-and-inherit model and directly inspired ecdysis. The other Rust option, shellflip, targets Oxy, Cloudflare's Rust proxy. shellflip assumes systemd and Tokio and focuses on transferring arbitrary application state between generations — more powerful for complex, stateful services, but more overhead for simpler cases.

Full documentation, API reference, and examples covering TCP listeners, Unix socket listeners, and systemd integration are available at docs.rs/ecdysis and in the repository examples directory.