Where IP selection comes from

When a query for a customer's proxied domain reaches Cloudflare's nameserver, the nameserver answers with an IP address of its choosing. Making Cloudflare authoritative for the domain means the nameserver is the authority for those queries; a recursive resolver such as 1.1.1.1 forwards the authoritative answer to the client, which uses it to connect to Cloudflare's global anycast network, where every data center advertises all of our addresses.

image3

If the customer has set up a static IP address, the choice is trivial: the nameserver returns that address. For every other proxied domain, the nameserver could return virtually any address Cloudflare owns and operates. The same address may answer queries for different domains, and different addresses may answer queries for the same domain. This supports resilience, and decoupling names from IP addresses improves flexibility.

So which IP should be returned for a proxied domain without a static IP? Cloudflare selects IP addresses to meet business objectives, for example to change the IP of a domain under attack, to send fractions of traffic to specific IPs for testing features or services, or to remap ("renumber") domains onto new address space.

Topaz programs as declarative objectives

To change nameserver behavior, an engineer encodes the desired DNS objective as a declarative Topaz program. The nameserver keeps the list of all such programs and, on receiving a query for a proxied domain, executes them in sequence until one returns an IP address, which it then hands to the resolver.

image1

Every Topaz program has three primary components:

  1. Match function: takes DNS query metadata (datacenter information, account information, and so on) and returns a boolean. If it evaluates to true for a query, the program's response function runs.
  2. Response function: given the same metadata, outputs a 3-tuple of IPv4 addresses, IPv6 addresses, and TTL, which is returned to the resolver.
  3. Configuration: a set of variables that parameterize the match and response functions. Referencing configuration variables separates a program's macro-level behavior from its nitty-gritty details (specific IPs, names), making the behavior easy to read at a glance.

As an example, a program can give every queried domain whose metadata field "tag1" equals "orange" a particular IP address:

- name: orange
  config: |
    (config
      ([desired_tag1 "orange"]
       [ipv4 (ipv4_address “192.0.2.3”)]
       [ipv6 (ipv6_address “2001:DB8:1:3”)]
       [t (ttl 300]))
  match: |
    (= query_domain_tag1 desired_tag1) 
  response: |
    (response (list ipv4) (list ipv6) t)

Configuration, match, and response functions are YAML strings, but more precisely they are topaz-lang expressions. Topaz-lang is the DSL built for expressing Topaz programs. It is based on Scheme but much simpler: dynamically typed, not Turing complete, and every expression evaluates to exactly one value (functions can throw errors). Operators cannot define functions in topaz-lang; new DSL functions are added only by writing functions in the host language, Go. The DSL provides basic types (numbers, lists, maps) plus Topaz-specific types such as IPv4/IPv6 addresses and TTLs.

  • The config holds four bindings from name to value: the string "orange" to desired_tag1; 192.0.2.3 to ipv4; 2001:DB8:1:3 to ipv6; and the TTL 300 (seconds, a topaz-lang type) to t.
  • The match expression must evaluate to a boolean. It can reference configuration values such as desired_tag1 and DNS query fields, all of which use the query_ prefix and are brought into scope at evaluation time. Here it compares desired_tag1 with the tag attached to the queried domain, query_domain_tag1.
  • The response expression evaluates to the special response type — a 3-tuple of an IPv4 address list, an IPv6 address list, and a TTL. Here it returns the configured address values and TTL.

Program order and conflict risk

All Topaz programs are encoded as YAML and live in one version-controlled file. Suppose the file holds only the orange program and a new team adds one that matches domains whose "tag1" is "orange" AND whose "tag2" is true:

- name: orange_and_true
  config: |
    (config
      ([desired_tag1 "orange"]
       [ipv4 (ipv4_address “192.0.2.2”)]
       [ipv6 (ipv6_address “2001:DB8:1:2”)]
       [t (ttl 300)]))
  match: |
    (and (= query_domain_tag1 desired_tag1)
         query_domain_tag2)
  response: |
    (response (list ipv4) (list ipv6) t)

The new orange_and_true program must sit either below or above orange in the file. Placing it after orange looks like this:

- name: orange
  config: …
  match: …
  response: …
- name: orange_and_true
  config: …
  match: …
  response: …

A program with narrower targeting

A third example: a team wants to test a modified CDN HTTP server on a small percentage of domains, only in a subset of data centers, spreading those queries over a specific IP prefix so that queries for the same domain get the same IP.

- name: purple
  config: |
    (config
      ([purple_datacenters (fetch_datacenters “purple”)]
       [percentage 10]
       [ipv4_prefix (ipv4_prefix “203.0.113.0/24”)]
       [ipv6_prefix (ipv6_prefix “2001:DB8:3::/48”)]))
  match: |
    (let ([rand (rand_gen (hash query_domain))])
      (and (member? purple_datacenters query_datacenter)
           (< (random_number (range 0 99) rand) percentage)))
  response: |
    (let ([hashed_domain (hash query_domain)]
          [ipv4_address (select_from ipv4_prefix hashed_domain)]
          [ipv6_address (select_from ipv6_prefix hashed_domain)])
      (response (list ipv4_address) (list ipv6_address) (ttl 1)))

The configuration binds purple_datacenters to (fetch_datacenters "purple"), a function that retrieves all Cloudflare data centers tagged "purple" through an internal HTTP API and returns them as a list. Next comes percentage, the fraction of traffic the program should act on, then two more bindings to IPv4 and IPv6 prefixes (using the built-in ipv4_prefix and ipv6_prefix types).

The match function uses a let form to define a local variable: a random number generator called rand, seeded with the hash of the queried domain name. Its conjunction checks two things: that the query landed in a data center tagged "purple", and that a random number between 0 and 99 from that seeded generator is below the configured percentage. Seeding with the domain means 10% of domains match; seeding with the query ID, by contrast, would make queries for the same domain behave differently. Together the conjuncts evaluate to true for 10% of domains queried in "purple" data centers.

The response function defines three local variables: a hash of the domain, an IPv4 address selected from the configured prefix, and an IPv6 address selected similarly. select_from always returns the same IP for the same prefix and hash, so a given domain always gets the same address — easing correlation of its queries — while different domains can receive different addresses within the prefix. The TTL returned is 1 second.

From YAML to the edge: the deployment pipeline

All Topaz programs live in one version-controlled YAML file. When an operator edits that file — adding, removing or modifying a program — the control plane works through four steps in order:

  1. Validate the programs for syntax errors.
  2. Finalize each program's configuration by evaluating every configuration binding and storing the result. Finalizing the purple program, for example, means evaluating fetch_datacenters and keeping the resulting list, so nameservers never have to fetch external data themselves.
  3. Verify the finalized programs.
  4. Distribute them across the network.

Distribution runs through QuickSilver, the edge key-value store: the control plane writes the program list there, and the Topaz service on each server notices the change and refreshes its own list.

Serving a query

When a DNS query arrives, the nameserver enriches it with metadata such as tags, then hands it to the Topaz service on the same machine over IPC. Topaz walks its program list and runs each program's match function through the topaz-lang interpreter, with the query in scope under the query_ prefix. The first match function that returns true wins: Topaz executes that program's response function and returns the resulting IP addresses and TTL, which the nameserver wraps in valid DNS format and sends back to the resolver.

Properties the verifier checks

Every program passes through the model-checker before it reaches the global network. If a program contains a bug, or if two programs conflict, the verifier raises an error. Three properties are checked.

Satisfiability

A program must be satisfiable: some DNS query has to make its match function return true. This catches internally inconsistent programs that could never match — a match expression like (and true false) has no satisfying query, so verification fails.

Reachability

A program must also be reachable: some DNS query must make it match given all programs ahead of it. This catches programs that are fully overshadowed by higher-priority ones. Take the ordering below:

- name: orange
  config: …
  match: (= query_domain_tag1 "orange")  
  response: …
- name: orange_and_true
  config: …
  match: (and (= query_domain_tag1 "orange") query_domain_tag2)
  response: …

Verification fails here because orange_and_true can never be reached. For any query where query_domain_tag1 is "orange", orange always matches regardless of metadata2. Swapping the two programs, as shown earlier, clears the error.

Conflicts between exclusive programs

The third check, and the most important, looks for program conflicts: queries that make two programs match at once. When one is found, verification fails and the offending query is printed, and operators must fix their programs before deployment. The check is scoped, however: it compares only programs explicitly marked exclusive, a marking operators apply when they need certainty that no other exclusive program can match the same queries.

Here is the corrected ordering with both programs marked exclusive:

- name: orange_and_true
  exclusive: true
  config: ...
  match: (and (= query_domain_tag1 "orange") query_domain_tag2)
  response: ...
- name: orange
  exclusive: true
  config: ...
  match: (= query_domain_tag1 "orange") 
  response: ...

The verifier now rejects the pair, identifying both programs as able to contradict each other and supplying a sample query as evidence:

Checking: no exclusive programs match the same queries: check FAILED!
Intersecting programs found:
programs "orange_and_true" and "orange" both match any query...
  to any domain...
    with tag1: "orange"
    with tag2: true

Both owning teams must settle the conflict before deployment, and the sample query gives them something concrete to work from. Removing exclusive from one program is the simplest resolution, accepting that the two cannot both be exclusive; order then decides priority. That outcome is legitimate. Topaz lets authors demand non-overlap via exclusive, but where non-overlap simply doesn't hold, at least priority is explicit. In practice, all exclusive programs sit at the top of the program file, which makes their interaction with non-exclusive programs easier to reason about.

Verification earns its keep twice over: it catches bugs through satisfiability and reachability, and it makes the consequences of a change visible. Operators get immediate feedback and must resolve conflicts before deployment rather than after an incident.

Semantic diffs

The verifier has gained an early-stage feature called semantic diffs. Even when a change passes verification, operators usually want to understand its impact. The verifier compares the old and new versions of the program file and looks for any query that matched program X before but matches a different program Y after, or the reverse. For instance, changing orange_and_true like this:

- name: orange_and_true
  config: …
  match: (and (= query_domain_tag1 "orange") (not query_domain_tag2))
  response: …

produces this output:

Generating a report to help you understand your changes...
NOTE: the queries below (if any) are just examples. Other such queries may exist.

* program "orange_and_true" now MATCHES any query...
  to any domain...
    with tag1: "orange"
    with tag2: false

The information is not exhaustive, but it shows operators whether a change does what they intend, before it ships. Diff capabilities are slated to expand.

Inside the verifier: Rosette, first-order logic and SMT

At a high level, the verifier establishes that the three properties hold for all possible DNS queries. A Satisfiability Modulo Theories (SMT) solver makes this tractable. It doesn't literally iterate over every query, but the result is equivalent: an exhaustive proof.

The implementation is written in Rosette, a solver-enhanced DSL hosted in Racket. That choice turns much of the work into engineering rather than formal logic — express your language's interpreter in Racket/Rosette and verification comes, in a sense, for free. The team wrote a topaz-lang interpreter in Racket and encoded the three properties with the Rosette DSL.

Rosette compiles the properties into first-order logic formulae that resemble algebra equations with unknowns. To check whether orange is reachable with orange_and_true ordered ahead of it, Rosette emits ((NOT orange_and_true.match) AND orange.match). The unknowns are the query parameters the match functions read, such as query_domain_tag1. Rosette passes the formula to an SMT solver such as Z3, which efficiently finds values for those parameters that make it true, and translates the solution back into a Racket data structure — here, a sample DNS query. Finding one proves orange reachable.

What verification costs

The main price is maintenance: the model checker's Racket interpreter must stay in lockstep with the production Go interpreter. If the two drift apart, the verifier stops detecting bugs accurately. New topaz-lang functions must also be verification-compatible.

Verifiability constrains what authors can write. Rosette handles only functions over integers and bit-vectors, so Topaz admits only operations convertible to those domains. The restriction sounds severe but goes a long way; strings are the hard part. Topaz rejects programs that manipulate substrings of the queried domain name, while simple operations on closed-set strings are fine — equality between domain names, for example, works because all strings reduce to a small set of integer-representable values. Because the verifier only ever examines match functions, only functions used there must be compatible; response code is unconstrained. Modeling details such as randomness posed further accuracy challenges.

Speed is a secondary cost. The seven existing programs verify against all three properties in about six seconds, which is fine since verification runs centrally at build time, only when programs change. Microbenchmarks put 50 programs at roughly 300 seconds — still acceptable, and optimizations are under investigation.

Status

What started as a research project now runs in production, formally verifying every change to the authoritative DNS behavior specified in Topaz. The SIGCOMM 2024 paper and the accompanying talk recording cover the system in more depth.