Accelerate building resiliency into systems with Cloudflare Workers

When email vendors go down: Adding a failover layer without rewriting services

Cloudflare's email notification pipeline depends on external Email Service Providers (ESPs) to reach customers with billing invoices, password resets, OTP logins, and certificate status updates. Multiple control-plane services dispatch messages through a vendor's HTTP Transmission API and SMTP endpoints, with exponential backoff retries protecting against message loss.

Delivery guarantees alone proved insufficient during a real incident. When the ESP's HTTP API and SMTP both became unavailable, messages queued for hours. While retry mechanisms ensured nothing was lost, time-critical emails like OTP logins have short validity windows. Delaying them is functionally equivalent to dropping them.

The failure exposed two gaps: there was no second vendor to route around the outage, and no mechanism to fail over quickly. Both needed to be addressed, and fast.

Why a second vendor is not a simple addition

Onboarding another ESP introduces an email deliverability complication. ISPs assign sending scores to organizations based on sender reputation, and new IP addresses are treated as suspicious until they earn trust. Mailbox providers build this reputation from engagement signals, e.g., whether recipients open and click messages. IP warm-up requires slowly ramping volume over days or weeks.

Using a new vendor only during failures would likely land messages in spam folders. To establish a positive reputation with the new vendor, the routing layer needs to send a constant stream of traffic to both ESPs under normal conditions. The solution is weighted traffic routing: an algorithm distributing email volume between vendors according to configured ratios, capable of shifting all traffic to the healthy vendor when one fails.

Where should routing logic live?

With the algorithm settled, the question became where to place it. Three designs were evaluated.

A shared library

The simplest approach: package the routing module as a library imported into every service. No architectural changes are required, but the cost is substantial. Every service would need code changes, end-to-end testing, and coordination across multiple teams. Library updates are slow to propagate, and there is no guarantee every team adopts the new version promptly. During an incident, some services could still be pointed at the failing vendor.

A dedicated microservice

Extracting the weighted-traffic routing into its own service gives cleaner ownership. Services simply update their base URL to point at the router, preserving the original ESP contract. Retry semantics live in one place, and failover configuration changes affect only the router.

The drawbacks are operational: building, testing, and shipping a new service takes time. The router becomes part of the request path, adding a network hop. Since all email traffic funnels through it, the service must be load-balanced and scaled to match incoming volume and availability requirements.

A Cloudflare Worker

Implementing the router as a Worker offers the same contract-preserving benefits as a microservice, with reduced operational overhead. Services need only environment variable changes in most cases. Failover is triggered by altering a Worker environment variable; once redeployed, the change takes effect across Cloudflare's network in milliseconds. Scaling and load balancing are handled automatically, and no containers need to be spun up or configured.

The remaining drawback of an added network hop is mitigated by available templates that make bootstrapping efficient.

Building the Worker-based router

The Worker sits between the control plane and the ESPs, proxying requests and streaming raw upstream responses. The internal services remain agnostic to the presence of multiple vendors.

addEventListener('fetch', async (event: FetchEvent) => {
try {
    const finalInstance = ratioBasedRandPicker(
      parseFloat(VENDOR1_TO_VENDOR2_RATIO),
      API_ENDPOINT_VENDOR1,
      API_ENDPOINT_VENDOR2,
    )
    // stream request and response
    return doRequest(request, finalInstance)
  } catch (err) {
       // handle errors
    }
})  

export const ratioBasedRandPicker = (
  ratio: number,
  VENDOR1: string,
  VENDOR2: string,
  generator: () => number = Math.random,
): string => {
  if (isNaN(ratio) || ratio < 0 || ratio > 1) throw new ConfigurationError(`invalid ratio ${ratio}`)
  return generator() < ratio ? VENDOR1 : VENDOR2  
}

Traffic is distributed using an environment variable that defines the ratio between vendor 1 and vendor 2. The Worker generates a random number between 0 and 1 with a uniform distribution. Setting the ratio to 1 routes everything to vendor 1, setting it to 0 routes everything to vendor 2, and intermediate values split traffic proportionally.

Configuration happens in one of two ways:

• Cloudflare Dashboard: edit the variable under Workers → Settings → Variables
• Wrangler: update the variable in wrangler.toml and run wrangler publish

Either method deploys the change to every Cloudflare data center nearly instantly, so all email traffic switches to the healthy vendor.

Failover in practice

After the router reached production, one vendor declared an outage. The response timeline shows how quickly the system recovered:

• 18:45 UTC: alert for slow delivery from provider 1; provider 2 unaffected
• 18:47 UTC: teams confirm outbound delays
• 18:47 UTC: failover decision made
• 18:50 UTC: router configured for 100% traffic to provider 2 and deployed

From that point, requests to provider 1 declined steadily while provider 2 traffic rose accordingly. All critical emails reached their destinations on time.

Next step: automated failover

Failover currently requires a manual step. The roadmap is to automate the process: when a vendor outage notification arrives, the Worker will shift traffic to the healthy vendor automatically.

The result satisfies the original goals: a second vendor improves availability, the Worker provides a fast configuration-based failover without code changes, and the implementation shipped quickly enough to prevent another major incident. Cloudflare's email systems are now resilient to individual ESP failures, with a central routing point that serves multiple internal teams.