How we migrated Dropbox from Nginx to Envoy

The case for replacing a decade-old proxy

Dropbox's legacy traffic infrastructure was built on Nginx with a largely static configuration rendered from Python2, Jinja2, and YAML. Every change required a full redeployment, while dynamic concerns like upstream management and stats exporting lived in Lua. Anything more complex was pushed down to the next proxy layer, a Go-based service called Bandaid.

Nginx held up well for nearly ten years, but it stopped aligning with where Dropbox engineering was heading. Internal and external APIs were moving from REST to gRPC, which demands transcoding support from the proxy layer. Protocol buffers had become the standard for service definitions and configuration. All software, regardless of language, was built and tested with Bazel. And engineers were increasingly contributing to essential open-source infrastructure projects.

Operationally, Nginx was also becoming costly to maintain. Configuration generation logic was spread too flexibly across YAML, Jinja2, and Python. Monitoring relied on a mix of Lua scripts, log parsing, and system-level checks. Third-party modules were increasingly necessary but hurt stability, performance, and the cost of future upgrades. Nginx's deployment and process management also diverged from the rest of Dropbox's services, depending heavily on external system configurations like syslog and logrotate rather than being self-contained.

For the first time in a decade, Dropbox began seriously evaluating a replacement.

Why not double down on Bandaid?

Bandaid, Dropbox's internal Go-based proxy, has deep integration with the company's infrastructure. It has access to the full ecosystem of internal Go libraries for monitoring, service discovery, rate limiting, and more. But it was not a viable candidate to replace Nginx for two primary reasons.

First, Go is more resource-intensive than C or C++. This matters especially at the edge, where Dropbox cannot easily auto-scale deployments. CPU overhead comes mainly from the garbage collector, the HTTP parser, and TLS, with the latter being less optimized than BoringSSL, which both Nginx and Envoy use. The goroutine-per-request model combined with GC overhead also drives up memory requirements significantly in high-connection services.

Second, Go's TLS stack lacks FIPS support. And Bandaid has no community outside Dropbox, meaning the company would be solely responsible for all feature development.

Given those constraints, Dropbox chose Envoy as the migration target.

Performance under production-style load

Both Nginx and Envoy use event-driven architectures. Nginx forks multiple worker processes and supports SO_REUSEPORT and EPOLLEXCLUSIVE along with worker-to-CPU pinning. Envoy runs on threads and also supports SO_REUSEPORT (with BPF filter support), but relies on libevent rather than using epoll extensions like EPOLLEXCLUSIVE.

The defining difference is blocking behavior. Nginx is not fully non-blocking: operations such as opening a file or writing logs can stall an event loop even with AIO and thread pools enabled. Envoy keeps every event-loop operation non-blocking, including logging.

We benchmarked both under representative workloads: high requests per second (RPS), high bandwidth, and mixed low-latency/high-bandwidth gRPC proxying. Overall throughput was similar, with two exceptions:

• Nginx showed higher long-tail latencies, mostly due to event-loop stalls under heavy I/O. With SO_REUSEPORT, a blocked worker can still accept connections on behalf of other workers, making the stalls worse.
• Our Lua-based stats collection slowed Nginx by 3x on high-RPS tests, driven by mutex-synchronized lua_shared_dict accesses.

We investigated replacing our stats approach with per-worker lockless counters in user space, similar to FreeBSD’s counter(9). The idea died on implementation cost: instrumenting Nginx internals such as error conditions would require maintaining a fork-sized patch across every upstream upgrade. Envoy has neither problem, so after migrating we could free up to 60% of the servers previously dedicated to Nginx.

Internal visibility

Non-commercial Nginx exposes seven metrics in its stub status module — clearly not sufficient. We piped around the gap with Lua: a log_by_lua handler emitting per-request stats — status codes, sizes, cache hits — to our monitoring system. Error classification was even more fragile: Nginx’s error.log had to be parsed to surface upstream, HTTP, Lua, and TLS errors separately, and we ran a dedicated exporter just to extract internal state like worker count, RSS/VMS sizes, and TLS certificate ages.

Envoy out of the box reports thousands of metrics in Prometheus exposition format, from version and uptime through memory allocator stats, connection-pool details, and timing histograms, with per-cluster, per-upstream, per-vhost, and per-listener breakdowns. The admin interface is operational gold:

• /logging toggles error log levels on the fly.
• /cpuprofiler, /heapprofiler, and /contention aid performance troubleshooting.
• /runtime_modify changes configuration parameters without a rollout, supporting feature gating.
• /certs, /clusters, and /config_dump expose additional structured state.

Envoy’s pluggable tracing integrates end-to-end from edge to application servers — useful both for our own traffic infrastructure team and for app developers who need full path latency. Nginx’s third-party OpenTracing module is not actively developed.

Access logging is another step forward: Envoy streams logs to a gRPC endpoint as defined by the Access Log Service (ALS) API. We no longer maintain syslog-to-hive bridges for our log pipeline.

Dataplane-control plane integration

Nginx configuration is deliberately static and file-based: config, TLS certs, access lists, logging targets. That simplicity is a strength at small scale, but as complexity grows, testing and standardization become bottlenecks.

Envoy mandates a more structured relationship between the dataplane and control plane through the xDS API family, which relies on protocol buffers and gRPC. The Universal Data Plane API (UDPA) ambitions read as overreach, but we already use ORCA for load testing and are evaluating UDPA for non-Envoy proxies such as our Katran eBPF/XDP load balancer.

Our homegrown control plane is a regular gRPC service. It adapts internal infrastructure facilities into a stable xDS surface, touched by HULK_*, at least conceptually free of cron, logrotate, syslog, and log-parsing hacks. The canonical example of how to integrate Envoy with a custom service discovery engine has considerable documentation; Istio and go-control-plane also exist as open-source reference implementations.

Typed configuration beats text templates

Nginx’s plain-text configuration is simple until it isn’t. Growth drives code generation, and complexity spreads across your templating stack. Prior to the migration we were using a mix of Python2, Jinja2, and YAML; YAML anchors, Jinja2 control flow, and Turing-complete Python all allowed substitution and logic with no clear data model. Worse, there was no formal schema: config that parsed cleanly could still fail semantically — buffer-size constraints, alignment limits, and interdependencies between variables only become visible by running nginx -t.

Envoy’s configuration is entirely protobuf-defined. This gives us a first-class data model with type safety for every value. Our new config generator uses protobuf-defined messages plus Python3 logic, with type annotations matched by mypy-protobuf. IDE type checking catches many defects before config is even rendered.

Circularly, value constraints are handled with protoc-gen-validate. For example, gzip window_bits may only take integer values from 9 through 15. Documentation also found a natural home next to the definition in the proto files.

For those nervous about losing schema-less flexibility, Envoy core developer Harvey Tuch discusses using google.protobuf.Struct and google.protobuf.Any to preserve extensibility in “Dynamic extensibility and Protocol Buffers.”

Beyond Lua and C modules

Nginx extension code often meant C modules written with low-level containers, manual memory management, and careful access to internal APIs — or third-party OpenResty libraries which we’ve used with log_by_lua and balancer_by_lua. Esoteric internals aside, the established sandlot-level Perl and JavaScript interfaces constrict extension to the content-response phase.

Envoy specializes in C++ plugin development:

• Clean, layered interfaces like the HTTP filter filter.h header reduce ramp-up and risk — modern C++ code is approachable, especially if you already live in Golang or Python.
• C++14 and the standard library give you templates, lambdas, and type-safe algorithms as basic tools, extended by Abseil with smarter mutexes and additional containers beyond the standard library.
• Our integration with the Vortex2 monitoring framework was implemented in about 200 lines of code, simply by exposing the Envoy::Stats::Sink and Envoy::Stats::Scope interfaces.

Envoy also regains ground on dynamic extensibility. Unlike OpenResty, the supported Lua integration is much more basic, so we steered clear and concentrated our effort on C++ filters. Far more interesting is Envoy’s support for WebAssembly (WASM) via the proxy-wasm specification: extensions compiled from any language targeting WASM execute inside a sandbox — near-native at typical 60-70% of C++`performance — and integrate with xDS for dynamic, tested rollout. WASM promises safe hosting for customer-authored edge extensions, plus opens native-adjacent execution to a broader ecosystem. Istio is investing heavily with WASM-based telemetry and the WebAssemblyHub sharing platform mentioned by their 2020 work. Dropbox does not yet use WebAssembly in production — we’re waiting on the Go SDK.

Better development experience

Nginx’s custom configure-and-make build system resists easy Bazel integration. Google did heavy lifting here, open-sourcing their Bazel-built Nginx project that packages BoringSSL, PCRE, ZLIB, and Brotli, and we had adopted it in Dropbox’s own monorepo. But test hygiene lagged: the project has a separate Perl-driven integration suite and unit tests at all were largely absent. We leaned on generated mock configs and Python test harnesses to exercise our Lua routes by hand.

Envoy and Bazel, by contrast, are natively compatible. Integrating it into Dropbox’s monorepo was almost anti-climactic thanks to Bazel’s external dependency mechanics, and we sync Envoy’s and UDPA’s protos regularly with copybara scripts. Unit testing comes to the forefront with googletest/GoogleMock mocks that eliminate the slow end-to-end setups for simple logic; drop-in test mocks and the integration test framework cover the rest. Open-source Envoy development sets a bar: 100% unit test coverage is required, verified against each pull request.

Sub-second unit test roundtrips unlock a compounding productivity loop: write more tests, get closer to that 100% coverage. Choosing between unit and integration tests flexibly balances coverage, velocity, and runtime cost, while quick Bazel, bazel test // waits even allow targeted micro-benchmark execution.

Bazel costs upfront but provides correctness-preserving incrementals, distributed builds and caching, and provides a programmable query interface to the dependency graph that enables metadata-aware linters, code generators, and, ultimately, internal deployment and vulnerability scanning tooling.

Good trade-offs have drawbridge issues

Nginx’s smaller code footprint and thin dependency list — typical builds link only zlib, TLS, and PCRE — helped achieve a clean 11-year security record of roughly 30 advisories. Envoy boots substantially more code, including rich third-party transport parsers, which inherently enlarges the attack surface. Mitigations show up as modern security practices baked into development: AddressSanitizer, ThreadSanitizer, MemorySanitizer, and fuzzing via OSS-Fuzz. Yet in practice the greater surface still leaks: Envoy closed 22 security advisories over the past two years, despite a formal security response plan with postmortems and participation in Google’s Vulnerability Reward Program.

Having measured both honestly, we’ve accepted the trade-off and look forward: Envoy’s typed, auditable data model and active integration improve our operability enough to outweigh the steeper security upkeep.

Why Envoy’s development model won us over

Nginx development is centralized: most of it happens behind closed doors, with external input limited to the nginx-devel mailing list and the official bug tracker. The result is visible in a project like HTTP/3, where F5’s own experimental implementation and Cloudflare’s separate QUIC fork exist as two divergent efforts with no shared path forward.

Envoy, by contrast, is coordinated out in the open through GitHub issues and pull requests, a mailing list, and regular community meetings, with further discussion on Slack. The HTTP/3 work is a concrete example of the difference. Envoy’s implementation builds on Chromium’s “quiche” library, but the design doc was published before any patches landed, progress is tracked publicly in a GitHub issue, and remaining work is tagged with “help wanted.”

That transparency had a direct payoff for Dropbox: it let us upstream a steady stream of small and medium changes — operational improvements, performance optimizations, new gRPC transcoding features, and load-balancing changes were all contributed back to the project.

Where the migration stands today

We have run Nginx and Envoy side by side for more than six months, shifting traffic incrementally via DNS. A wide range of workloads has already moved over:

• Ingress high-throughput services: all file data for the Dropbox desktop client is served end to end over gRPC through Envoy. Users saw a slight performance improvement from better connection reuse at the edge.
• Ingress high-RPS services: all file metadata for the desktop client, benefitting from the same end-to-end gRPC path plus removal of the connection pool.
• Notification and telemetry services: these hold millions of HTTP connections for real-time notifications, and can now use streaming gRPC instead of an expensive long-poll model.
• Mixed high-throughput/high-RPS services: public API traffic for both metadata and data, opening the door to public gRPC APIs or transcoding existing REST endpoints at the edge.
• Egress high-throughput proxies: Dropbox-to-AWS communication, mostly S3, with the goal of eventually removing all Squid proxies from production.

The last major target is www.dropbox.com itself. Once that moves, we can start decommissioning edge Nginx deployments.

Compatibility gaps and fixes

The migration was not flawless, but it caused no notable outages. The hardest part was our API services, because the public API serves everything from shell scripts using curl/wget to embedded devices with custom HTTP/1.0 stacks. Nginx is the de-facto standard, so many HTTP libraries implicitly depend on some of its behaviors. We found a number of Envoy inconsistencies and bugs, but all were resolved and upstreamed quickly with community help.

Some of the more notable non-RFC behaviors we had to handle:

• URL slash merging: Nginx enables slash normalization and merging by default; Envoy lacked the latter. We contributed a patch adding an opt-in merge_slashes option to the HTTP connection manager.
• Ports in virtual host names: Nginx accepts the Host header with or without a port. A few API users relied on this. After working around it by duplicating vhost configs, we upstreamed an option to ignore the matching port — strip_matching_host_port.
• Transfer-Encoding case sensitivity: A small subset of clients sent Transfer-Encoding: Chunked with a capital “C.” That is technically valid per RFC7230, and the fix was a trivial upstream submission.
• Requests with both Content-Length and Transfer-Encoding: chunked: Nginx allowed these; Envoy rejected them. RFC7230 is ambiguous here — web servers should error such requests as potential request smuggling, but a later sentence says proxies should strip Content-Length and forward. We extended http-parser to let library users opt into supporting these requests and are adding support to Envoy itself.

Two common configuration issues are worth flagging for anyone planning an edge Envoy rollout:

• Circuit-breaker misconfiguration: default circuit-breaking limits are tight. For inbound proxies, especially in mixed HTTP/1 and HTTP/2 environments, improper settings caused unexpected downtime during traffic spikes and backend outages. Consider relaxing them outside a mesh context.
• Buffering: Nginx can buffer request bodies to disk, letting it convert chunked transfer encoding into requests with Content-Length for legacy HTTP/1.0 backends. Envoy’s buffer filter can only use memory, which caps how much we can buffer.

For anyone evaluating Envoy at the edge, the official “Configuring Envoy as an edge proxy” guide is worth reading; it covers security and resource limits relevant to the most exposed part of your infrastructure.

Roadmap

• HTTP/3: browser support is becoming generally available, and Envoy’s experimental implementation is in place. After we upgrade the Linux kernel for UDP acceleration, we will experiment with QUIC at the edge.
• Internal xDS-based load balancing: we are evaluating the combination of the Load Reporting service (LRS) and the Endpoint Discovery Service (EDS) as the foundation for a common load-aware load balancer for Envoy and gRPC.
• WASM extensions: once the Go proxy-wasm SDK is available, we can write Envoy extensions in Go and leverage our internal Go libraries.
• Bandaid replacement: unifying all proxy layers under a single data plane is compelling, but would require migrating many of Bandaid’s load-balancing features to Envoy. It is the long-term plan.
• Envoy mobile: we eventually want Envoy in our mobile apps to support a single, unified traffic stack with modern capabilities such as HTTP/3, gRPC, and TLS 1.3 across all platforms.

Acknowledgements

This migration was a broad cross-team effort. The Traffic and Runtime teams led it, with substantial contributions from Agata Cieplik, Jeffrey Gensler, Konstantin Belyalov, Louis Opter, Naphat Sanguansin, Nikita V. Shirokov, Utsav Shah, and Yi-Shu Tai — plus the Envoy community throughout. We also recognize Runtime team tech lead Ruslan Nigmatullin, whose work as Envoy evangelist, MVP author, and primary software engineering driver made the project possible.

Why the migration to Envoy matters for the Edge

Dropbox’s Edge network handles globally distributed traffic at terabit scale and processes millions of requests per second. Managing that load requires a traffic infrastructure that can scale cleanly with feature velocity and operational demands. The move from Nginx to Envoy was driven by the need for a more programmable, observable, and maintainable proxy layer that could keep pace with internal growth.

The problems with scaling Nginx

Nginx served Dropbox well for years, but as the requirements around dynamic routing, service discovery, and request-level control grew, certain limitations became unavoidable. The team needed deeper integration points with internal systems and richer data for debugging and performance analysis. Nginx’s module model and configuration paradigm became a bottleneck when trying to implement fine-grained traffic policies and metrics at scale.

Additionally, operating Nginx across the Edge involved substantial manual toil. Configuration management, reload cycles, and per-host customization were all pain points that multiplied with every new service or region.

Why Envoy fit the long-term architecture

Envoy was chosen for its modern, API-driven design. Its support for dynamic configuration via xDS protocols meant that routing rules, clusters, and listeners could be updated without process reloads. This was a fundamental shift from Nginx’s static config plus reload model, and it aligned with Dropbox’s push toward more automated infrastructure.

The proxy also provided first-class support for observability. Structured logging, distributed tracing integration, and rich statistics exposed through its admin interface and metrics endpoints gave the team visibility that Nginx required custom modules to approximate.

For the Edge team, Envoy’s filter chain architecture was a major advantage. It allowed the team to build reusable, composable middleware for authentication, rate limiting, and request transformation — all expressed in code rather than haphazard config snippets. This opened the door to standardizing edge behaviors across every entry point while reducing the risk of misconfiguration.

Migration strategy and operational impact

The migration was not a single cutover. The team incrementally shifted workloads from Nginx to Envoy, starting with lower-risk traffic and then expanding to more sensitive paths as confidence grew. This allowed validation of performance, correctness, and failure modes under real conditions.

A phased approach also meant that feature gaps could be identified early. Some Nginx-specific behaviors had to be replicated as Envoy filters or Lua scripts to avoid regressions. In several cases, the process highlighted where Nginx’s flexibility had masked inconsistencies that needed to be resolved before the switch.

Post-migration, the operational burden decreased noticeably. Configuration pushes through the control plane replaced error-prone reload cycles, and the richer telemetry reduced time spent correlating logs across hosts. The ability to run canary analyses and rollback at the control plane level further lowered the risk of edge changes affecting user-facing traffic.

Next steps for the Traffic team

With Envoy now handling the core edge proxy responsibilities, Dropbox is positioned to invest more in control plane features and further automation. The team’s focus is on refining dynamic routing policies and expanding the use of advanced observability to proactively detect anomalies before they impact users.

Improving the developer experience for service owners is another priority. By exposing more configuration via API and reducing the need for manual intervention, the team intends to make the edge a more self-service platform for internal teams.

The migration is a foundation, not a finish line. Envoy gives Dropbox a platform to iterate on traffic management without the constraints that eventually made Nginx’s model difficult to sustain at global scale.