Rearchitecting a DNS resolver for scale
Cloudflare launched its 1.1.1.1 public DNS resolver in April 2018, and has since layered on features including a debug page, global cache purge, 0 TTL support for zones on Cloudflare, end-to-end encrypted upstream TLS, and 1.1.1.1 for Families. Underneath those products sits Knot Resolver, chosen for its battle-tested DNS resolution and DNSSEC validation. That let the team focus on building Cloudflare-specific functionality rather than the DNS protocol itself.
Knot Resolver's Lua-based plugin system made it easy to extend core behavior for requirements like DoH/DoT termination, logging, BPF-based attack mitigation, cache sharing, and iteration logic overrides. But as traffic climbed, the architecture began to show strain. The team identified three problem areas that would drive a redesign: blocking I/O in plugins, cache efficiency, and plugin isolation.
The data center context
Every Cloudflare server runs the same software stack; only the configuration differs. DNS requests are load-balanced to individual servers inside a data center by Unimog, while DoH traffic terminates at Cloudflare's TLS terminator. Small configuration payloads are distributed worldwide via Quicksilver. This uniformity keeps fleet maintenance tractable, and it means the resolver process, kresd, can focus purely on resolving queries rather than transport details.
Three constraints emerged from this setup. First, the resolver is single-threaded, so any blocking operation in a callback stalls all concurrent requests. Second, requests for a domain can land on any node, so caches are redundant across servers and can be evicted independently. Third, the growing list of Lua modules shares a single Lua state, making failures hard to isolate and debug.
The event loop bottleneck
Knot Resolver plugins are callback-based modules invoked at defined points in request processing. They can inspect, modify, or generate requests and responses. By design, callbacks should be quick, since the single-threaded event loop serves many requests simultaneously. Even one slow callback blocks everything else on that worker.
That design held up until the resolver needed to perform blocking operations—for example, retrieving data from Quicksilver before replying to a client. That kind of synchronous fetch inside a callback could hold the entire loop hostage.
Wasted cache space
Because a given domain's queries can hit any server in a data center, each node would independently resolve the same name unless caches were shared. The team built a cache module that multicasts newly added cache entries to all nodes in the data center, letting them update local caches without repeating upstream lookups.
The default cache backend, LMDB, was adequate for smaller deployments but not for Cloudflare's scale. It lacks TTL awareness, popularity tracking, or intelligent eviction—when full, it simply drops everything and restarts. Zone enumeration attacks could fill the cache with junk that would never be requested again. The multicast module compounded the problem by amplifying that low-value data to every node, pushing all caches to their high-water mark simultaneously, which produced latency spikes as all nodes dropped and rebuilt caches around the same time.
Isolation gap
As modules multiplied, debugging became harder because all of them share one Lua state. A misbehaving module could corrupt or starve others. Failures such as too many coroutines or out-of-memory conditions might crash the process—or produce nearly unreadable stack traces. Forcibly tearing down or upgrading a running module was also difficult since state persists both in the Lua runtime and in FFI bindings, with no memory safety guarantees.
A closer look inside the async core
The first serious attempt at a replacement resolver wrapped Knot Resolver’s core with a thin Rust service built on a modified edgedns. That approach struggled with constant conversions between storage and C/FFI types, plus ABI quirks — cached records, for example, were expected to stay immutable until the end of a read transaction. The experiment still paid off: it taught the team how to design the host/guest boundary between a service and an embedded resolver library.
Later iterations swapped in a new recursive library built on tokio, which provided a thread pool for mixed blocking and non-blocking work. As futures combinators grew tedious, development moved onto nightly Rust to use async/await before it stabilized in Rust 1.39. Once stable, request processing became more readable, and tasks could run concurrently with work-stealing across threads. That avoided the earlier single-event-loop problem where one slow request blocked everything else.
The resulting platform is called BigPineapple. Inbound requests arrive at a server module, get validated, and are transformed into unified frame streams. A set of workers then resolve each frame, checking the cache module first and falling back to a recursor module that iterates the query. The recursor itself performs no I/O; it delegates sub-tasks to a conductor module, which handles upstream queries. A sandbox module lets plugins hook into the process along the way.
Decoupling inbound and outbound I/O
The frame abstraction normalizes UDP packets, TCP segments, and HTTP payloads into one representation of a DNS message with metadata. That lets the server enforce fairness and pacing across frame sources without protocol-specific logic. A key lesson from earlier builds: for a public service, pacing clients evenly matters more than peak I/O performance, because cache hits and misses cost vastly different amounts of time and both consume upstream authoritative server resources.
On the outbound side, the conductor tracks upstream metrics such as RTT and quality of service. It decides which nameserver to connect to, which protocol to use, and when to retry if a UDP packet may have been lost. Requests are deduplicated first: within a small window, identical queries collapse into one wire request, with the others queued behind it. This is especially useful when a popular cache entry expires.
The conductor can also route through another Cloudflare data center using Argo Smart Routing. The connection instructor generates parameters for the I/O executor, which then opens a direct connection or follows the alternate path.
Cache without the multicast
BigPineapple’s cache no longer uses a KV store. It’s built on an adaptive replacement cache data structure, which evicts less popular entries progressively and resists scans. The real change is at the data center level: rather than duplicating the cache across every node via multicast, nodes now relay queries to each other using consistent hashing. Queries for the same registered domain consistently land on the same subset of healthy nodes, raising the cache hit ratio and helping the infrastructure cache that tracks nameserver performance.
An async recursive library
The recursor is written as an async function that produces a response but never touches the network. Instead it takes an Exchanger trait — a Rust interface that knows how to exchange DNS messages with upstream servers. The logic reads sequentially: look up the closest cached delegation, and if it’s not final, await a response from upstream before proceeding. Because waiting is decoupled from recursion logic, tests can plug in a mock exchanger, and DNSSEC validation code stays readable instead of scattering across callbacks.
async fn resolve(Request, Exchanger) → Result<Response>;Writing a recursive resolver from scratch was hard, partly due to DNSSEC complexity and partly due to workarounds for RFC-incompatible servers, forwarders, and firewalls. The team ported deckard to Rust for testing. When the new library was ready, it first ran in “shadow” mode, comparing its answers against production traffic. For a recursive service, this produces false positives — authoritative servers often give different answers for the same query due to localization and load balancing. After a public test endpoint launched in December 2019, production endpoints migrated gradually, with edge cases in DNSSEC validation continuing to surface but becoming far easier to reproduce and fix.
Plugins move into a Wasm sandbox
The old plugin system ran Lua in the same memory space as the resolver. That allowed zero-copy cache reads but also let modules read uninitialized memory, call host ABIs with wrong signatures, or block on sockets with no restrictions from the host. After considering JavaScript and native modules, the team settled on WebAssembly, which runs programs in an isolated memory space and lets plugins be written in the same language as the service.
BigPineapple’s runtime is powered by Wasmer, chosen over Wasmtime and WAVM for simplicity of use. Each module runs in its own instance with isolated memory and a signal trap. Multiple instances of a module can run concurrently, and apps can be hot-swapped between instances without dropping a request. Because Wasm programs are distributed via Quicksilver, functionality can change worldwide within seconds.
The sandbox model introduces a few terms:
- Host: the program running the Wasm runtime, with full control over guest apps.
- Guest application: the Wasm program in the sandbox, which can only reach its own memory and imported host calls.
- Host call: functions the host exports for the guest; the only way out of the sandbox.
- Guest runtime: a library implementing common interfaces so apps can use async, socket, log, and tracing without details.
A guest app starts with a start function, called by the host on load, much like a regular executable’s entrypoint. That function typically registers callbacks for different query phases — cache lookup, delegation chain resolution — and may spawn background tasks for metrics or pre-fetching. The phase callbacks are closures, so they cannot be exported directly. Instead the guest calls a host call with the callback address, e.g. hostcall_register_callback(pre_cache, #30987). When the host needs to invoke it, it uses a trampoline function, trampoline_call(#30987), because the raw pointer lives in guest memory.
Isolation overhead and its workarounds
The sandbox’s memory isolation adds cost. Guest apps cannot read host memory, so data passed in normally requires a copy. For query processing, which reads request data on every call, that copy would be expensive. Instead, after instantiation, the guest pre-allocates a memory region that the host maps to shared memory with common request data. Once set up, the guest reads directly from the overlay with no copy needed.
Crypto presents another issue. The WebAssembly instruction set lacks AES and SHA-2 primitives, so a modern protocol like oDoH cannot benefit from host hardware inside the sandbox. With WASI-crypto still in progress, the team delegates HPKE operations to the host via host calls, already seeing a 4x performance improvement over in-Wasm execution.
Async inside the sandbox
To keep sandboxed callbacks from blocking, BigPineapple uses Rust’s async framework to bound how long a callback can occupy a thread. A Future in Rust needs a pollable function driving state transitions and a waker to trigger re-polling when an external event occurs. In the sandbox, host calls implement I/O: the guest calls hostcall_socket_open to get a handle, which maps to a descriptor on the host side, and later issues read or write calls.
fn poll(&mut self, wake: fn()) -> Poll {
match hostcall_socket_read(self.sock, self.buffer) {
HostOk => Poll::Ready,
HostEof => Poll::Pending,
}
}Inside a host call, the host not only performs the I/O but also registers the guest’s current waker. When the socket becomes ready, the host wakes the corresponding guest task via the trampoline. The same mechanism handles waiting between guest tasks, such as an async mutex. All of this is wrapped in the guest runtime, presenting ordinary async functions to apps without exposing the underlying machinery.
Growing the Platform
The architecture behind 1.1.1.1 remains a work in progress, with the system designed to accommodate new features and services. Today, a number of Cloudflare offerings run as guest applications on this underlying framework, including 1.1.1.1 for Families, AS112, and Gateway DNS. This modular approach allows the team to integrate new technologies and extend the system's capabilities.
Cloudflare is actively seeking input on potential future directions for the platform. Feedback can be shared through the community forums or via direct email to the resolver team.
