A syslog-ng to OpenTelemetry swap at Cloudflare scale
Cloudflare’s logging pipeline handles millions of log events per second across every server the company operates. The team recently replaced the syslog-ng daemons running on each server with OpenTelemetry Collectors. The goal was a transparent swap: all existing downstream systems should keep working without modification.
The migration touched edge and core data centers differently, required four custom internal components, and surfaced several lessons about what works — and what doesn’t — when replacing a piece of infrastructure this central.
Why move off syslog-ng
The reasons for the shift were practical rather than performance-driven. syslog-ng is written in C, which isn’t a core competency for the team maintaining the pipeline. The OpenTelemetry Collector is written in Go, opening up contributions to more engineers.
There were also operational benefits. Building syslog-ng against Cloudflare’s internal Post-Quantum cryptography libraries required maintaining a brittle C build chain; the Go build model is already optimized for that. OpenTelemetry Collectors come with built-in Prometheus metrics support, giving the team deeper telemetry about the collectors themselves — meta-observability that syslog-ng couldn’t easily provide. And since Cloudflare already runs OpenTelemetry Collectors for tracing, consolidating onto one daemon for all telemetry types reduces cognitive load.
Building an internal distribution
The upstream contrib repository has many useful components, but it became clear early that the team would need custom ones. That meant building a custom distribution using OCB (OpenTelemetry Collector Builder). The OCB configuration file was eventually templated to automatically include all internal components.
Four internal components were built for the initial version.
cfjs1exporter
The existing pipeline uses a line format called “cfjs1”: a JSON-encoded log with a format field determining the log type, and a wrapper field containing the structured JSON log body. The field name of the wrapper changes based on the format field, and together they determine which Kafka topic a receiver places the message into.
Since downstream systems weren’t changing, the collector had to support this format. The team built the cfjs1exporter modeled on the contrib repository’s syslogexporter. Long-term, the goal is to move to OTLP (OpenTelemetry Protocol) as the line format, which would let them drop the custom exporter and use open standards.
fileexporter fork
The upstream file exporter only supports JSON and Protobuf formats. Cloudflare needed plain text and syslog output, so they forked it internally. The plain text formatter writes log message bodies to a file with newline delimiters; the syslog formatter writes RFC 5424 formatted messages.
The fork also adds custom permissions support. The upstream exporter has two modes: a standard mode without compression or rotation, and an advanced mode using lumberjack for rotation. Features don’t behave consistently across modes. Community response to adding permissions has been split — open to the idea in native handling, but lumberjack maintainers oppose it — which pushed the team to implement it themselves.
Upstreaming these changes would be welcome, though it’s not clear how custom marshaller support would work with OCB. The team would prefer removing the need to maintain the fork.
externaljsonprocessor
A custom processor handles enrichment from external sources. It periodically queries HTTP endpoints or runs specific commands, adding the results as fields to all logs flowing through it. Cloudflare uses this to fetch data like data center status or systemd unit health, giving engineers more filtering options — for example, excluding logs from data centers that shouldn’t receive customer traffic or from servers disabled for maintenance.
This approach updates those values much faster than the standard three-hour cadence of salt-based configuration updates, which matters for fields that change quickly during network operations.
ratelimit processor
A replacement for syslog-ng’s ratelimit filter (also contributed upstream by Cloudflare) was needed. The ratelimit processor applies rate limits based on a specific field of a log message, dropping messages that exceed a limit with an optional burst limit. Cloudflare applies these over the service field so no single service can degrade log collection for others.
Though there has been upstream discussion of similar components, nothing fit. Intentional data loss during rate limiting — which is a feature in Cloudflare’s case — is hard to generalize into something broadly applicable.
Deploying edge vs. core
Deployment split into two paths. Core data centers host a small number of servers with diverse workloads — PostgreSQL, ElasticSearch, Kubernetes, and everything in between. Edge data centers are homogenous, with a much larger number of servers all running the same services.
Both use salt for configuration. The first step was writing salt states to install the collector and write configurations to disk, plus temporary migration pieces to disable syslog-ng and start the collector — and the inverse for rollback.
For edge data centers, once configurations were written, rollout was mostly gradual across servers. Since edge servers run identical services, gaining confidence in one set of configurations meant it became a slow, monitored rollout. There were several false starts requiring more instrumentation in the cfjs1exporter to handle niche services and general internet issues.
Core data centers needed a hands-on approach. Many core services have custom syslog-ng configurations — PostgreSQL servers have custom audit log handling, Kubernetes servers have custom contour ingress and error log handling. Each role with a custom config had to be manually onboarded, with extensive testing on designated canary nodes to validate the configurations.
What the Failure Modes Taught Us
The rollout surfaced two distinct failure modes that shaped how we hardened the new pipeline. The first appeared during scheduled chaos testing in our core data centers. When the primary central logging server was taken down, the cfjs1 exporter failed to detect the loss of connectivity. As a result, the collector never triggered failover to the secondary server. Its log buffer filled to capacity, and it stopped consuming from its receivers.
Journal receivers were immune to this problem because journald buffers logs before the collector reads them. Named pipe receivers were not so lucky. With consumption halted, services writing to the pipes in blocking mode saw their threads stall. Our legacy syslog-ng deployment handled this scenario with a monit script that periodically killed connections to the central receivers. For the collector, we chose a more explicit fix: tighter timeouts in our exporter and modifications to the upstream failover connector so it responds correctly to partial failures rather than assuming a binary up/down state.
The second issue was a cutover delay introduced during the live migration. We run dynamic edge workloads under Nomad using a custom driver that writes container logs to a named pipe. Our configuration management initially stopped syslog-ng and started the OpenTelemetry Collector as two separate steps. In the gap between those steps, nothing drained the named pipe. Services writing in blocking mode—including the Nomad driver itself—timed out and, in the driver's case, rescheduled the affected containers.
We caught this early in testing and changed the rollout mechanism. Instead of two discrete Salt actions, we used Salt to schedule a systemd oneshot service that stopped syslog-ng and started the collector simultaneously. That collapsed the downtime window to near zero and eliminated the starvation condition.
Where the Pipeline Goes From Here
Completing the migration of a component that had gone largely unchanged for nearly five years came with friction, but the new architecture removes the constraints we were living under. With the collectors in place, the roadmap for the logging pipeline now includes:
- Improved log sampling strategies, including tail sampling
- Better visibility for engineering teams into their telemetry production
- Adoption of OTLP as the line protocol
- Upstreaming select custom components we built during the migration
