Using Apache Kafka to process 1 trillion inter-service messages

One Trillion Kafka Messages: Cloudflare’s Messagebus Architecture

Cloudflare has relied on Apache Kafka for inter-service communication since 2014. What began as a modest deployment has grown into 14 distinct clusters across multiple data centers, totaling roughly 330 nodes. Over the past eight years, these clusters have processed more than one trillion messages—a figure that excludes the dedicated clusters powering customer-facing analytics dashboards, which alone handle over a trillion messages each day.

Kafka serves as the backbone for decoupling microservices at Cloudflare. When resources are created, changed, or deleted, services communicate these events through a common, fault-tolerant data format. This architectural choice is a key enabler for multiple engineering teams shipping features concurrently without stepping on each other’s toes.

The Application Services team owns the charter to make this ecosystem approachable. Their goal: provide tools and services that let product teams focus on delivering customer value rather than wrestling with distributed system plumbing.

Lowering the Barrier to Entry

The most general-purpose cluster, aptly named Messagebus, was designed to solve three problems: prevent data silos, enable near-zero-cost service-to-service communication, and encourage self-documenting message formats that eliminate stale documentation.

To drive adoption, the team built two internal projects. The first, Messagebus-Client, is a Go library wrapping Shopify Sarama with opinionated configuration and automated mTLS certificate rotation.

This convenience came with a hidden cost. By abstracting core Kafka concepts, the library made it too easy for teams to make innocuous-looking configuration changes with outsized consequences. One such change caused significant partition skew—where the majority of messages landed on a single partition, stalling real-time processing. Kafka’s constraint of one consumer per partition meant teams couldn’t simply scale horizontally to recover.

The incident highlighted an important pre-production step: back-of-the-napkin throughput calculations. Without them, teams could find themselves needing to add partitions after deployment, a non-trivial operation. The library has since been amended to make such misconfigurations less likely.

The Connector Framework

Building on the client’s success, the team identified common patterns in how services consumed and produced data. This led to the connector framework, which lets engineers spin up services that read from a system of record and push data to Kafka, Cloudflare’s own Quicksilver, or other targets. A Cookiecutter-based CLI generates deployable services from a few parameters.

Configuration is handled entirely through environment variables. Simple pipelines work out of the box, while extensibility comes from satisfying Go interfaces and “registering” custom readers, writers, and transformations. For instance, setting these variables:

READER=kafka
TRANSFORMATIONS=topic_router:topic1,topic2|pf_edge
WRITER=quicksilver

• Reads messages from Kafka topics topic1 and topic2;
• Transforms each message via the pf_edge function, converting a Kafka protobuf into a Quicksilver request;
• Writes results to Quicksilver.

Connectors ship with pre-configured metrics and alerts, so teams gain production confidence without building their own monitoring stack.

The Communication Preferences Service (CPS) demonstrates this pattern in action. When a user updates marketing opt-ins or language preferences on cloudflare.com, CPS uses a connector to read from Messagebus and propagate those changes across all relevant downstream systems.

Strict Schemas from Day One

Alongside the client library, the Messagebus Schema repository serves as the registry for all message types on the Messagebus cluster. Protobuf was chosen over JSON for several compelling reasons: type safety, enforced forward and backward compatibility, substantially smaller message sizes, multi-language code generation, and human-readable definitions.

Before a schema is merged, heavy commentary is encouraged. Once merged, prototool runs breaking-change detection, enforces style rules, and generates code for Go and Rust (with more languages easy to add).

The schema repository also maps each proto message to its owning team and that team’s internal chat channel. When issues arise, escalation to the right engineers is immediate.

A deliberate design decision—one proto message per topic—simplified adoption but created a sprawl of topics. Each topic gets multiple partitions with a replication factor of at least three, leaving headroom to optimize compute for lower-throughput topics.

Observability by Default

Cloudflare’s decoupled engineering model depends on teams being able to observe their Kafka footprint independently. Infrastructure is managed via Salt with a GitOps model, where the repository is the source of truth. To create a topic, engineers submit a pull request adding a few lines of YAML.

Upon merge, two things happen automatically: the topic is created, and a high-lag alert is provisioned. Lag—the gap between the last committed consumer offset and the last produced offset—is a powerful proxy for detecting a range of failures, including:

• A consumer that isn’t running.
• A consumer that can’t keep pace with production rates or an anomalous production spike.
• A misbehaving consumer that isn’t acknowledging messages.

Each new topic also gets an auto-generated Grafana dashboard showing production rate, consumption rate, and partition skew by producer and consumer. Alert messages include direct links to these dashboards, minimizing time-to-diagnosis.

The Messagebus-Client itself exposes default metrics that teams can extend:

Producers:

• Messages successfully delivered.
• Messages failed to deliver.

Consumers:

• Messages successfully consumed.
• Message consumption errors.

Teams use these metrics for throughput-change alerts or to detect idle topics where no messages flow within a timeframe.

Real-World Example: Alert Notification System

The Alert Notification System (ANS) powers the Notifications tab in the Cloudflare dashboard. In the past year, new alert and policy types have shipped frequently because ANS is built on the Messagebus foundations.

Enabling a new alert type requires three steps:

1. Add an entry to ANS’s configuration YAML, validated with CUE lang during CI.
2. Import Messagebus-Client into the service’s codebase.
3. Emit a message to the alert topic when the relevant event occurs.

That’s the entire integration. The producing team immediately gains support for granular customer alerting policies, dispatch via Slack, Google Chat, custom webhooks, PagerDuty, and email through both API and dashboard, plus managed retries and dead-letter handling.

What’s Next

Kafka usage at Cloudflare continues to grow, and the team remains committed to refining the Messagebus toolchain—focusing on usability, customizability, and observability. Driven by engineer feedback, the Messagebus-Client is now in its fifth major version. An experiment is underway to abstract Kafka entirely, letting teams stream messages via gRPC directly to Kafka—a project whose outcome will be the subject of a future post.