Logpush exports Cloudflare logs to SIEMs, analytics platforms and object storage in near real time, delivering batches as quickly as possible. Before those records reach a destination, teams often need to strip health-check traffic, omit sensitive fields or reshape records for dashboards and investigations — work that has typically required a separate processing service to deploy, scale and monitor. In addition to being made available on all self-serve plans during Birthday Week, Transformers reached general availability, moving that processing into the export path itself.
A Transformer is a SQL query defined against a specific Logpush dataset. It names fields from that dataset and determines which records and fields Logpush exports. Logpush runs the query on every batch before export: rows that fail the WHERE clause are dropped, and for the rest the SELECT output becomes the exported record. Results leave in NDJSON format. Teams create, preview and manage Transformers in Transformer Studio in the dashboard or through the API, save them, and attach them to a compatible NDJSON Logpush job that shares the same dataset. Every update produces a new version, so the history of a Transformer stays reviewable.
Filtering before delivery also cuts cost, since teams currently pay to export, ingest and store records they discard on arrival. A team sourcing logs from several CDNs, for instance, can inject a static provider field to normalize records upstream of its analytics platform.
Execution engine and runtime design
Bufstream, Cloudflare's Logpush delivery service, pushes more than 110 million records per second to customer destinations, part of a pipeline ingesting over one billion events per second. It runs on hundreds of Kubernetes pods, each claiming a time slice from Buftee, the in-house distributed log-based message broker; work spreads horizontally across jobs, with volume following a Pareto-like distribution where most jobs are small and a few process millions of logs each second. That set the bar for Transformers: reliable delivery, high throughput and predictable resource use, with no new failure scenarios in the export hot path.
An early prototype ran customer-defined WebAssembly Transformers in Cloudflare Workers, but invoking a Worker would insert a network call into the export path — a remote dependency whose transient timeouts and retries could dent delivery throughput and latency. Transformer execution stayed in-process instead, and of the SQL engines evaluated, Cloudflare chose Apache DataFusion, already used by Cloudflare Basin.
DataFusion is a Rust query engine built on Apache Arrow, with a SQL planner and a vectorized, columnar execution engine. Against a synthetic 5,000-row http_requests batch, a prepared Transformer executed from a single Go goroutine in about 3 ms — roughly 1.6 million rows per second. Attaching a Transformer adds CPU work to a job's processing path, so Bufstream parallelizes transforms per job to let high-volume jobs run batches concurrently, reusing the service's existing scheduling, batching and destination controls rather than standing up a separate transformation tier.
Moving typed batches across the Go/Rust boundary
Bufstream is written in Go; DataFusion is written in Rust. Bridging them efficiently is the core of the in-process pipeline. Bufstream takes compressed Logpush batches containing Cap'n Proto records from Buftee, decompresses them, parses the eligible logs and builds a typed Apache Arrow RecordBatch before running the transform.
Arrow is a language-independent columnar in-memory format for efficient analytics, representing flat and nested data as typed, contiguous columns. Arrow IPC gives the two runtimes a standard exchange format across the FFI boundary, exposed through a C ABI and called from Go via cgo. Because the Go and Rust runtimes do not share ownership of transform data, Go passes a pointer to its input Arrow IPC buffer to Rust only for the duration of the cgo call, and Rust retains no pointer afterward. Rust owns the transformed output buffer; Go copies it into Go-managed memory and then explicitly releases the Rust allocation, keeping ownership and lifetimes explicit to avoid cross-runtime memory bugs.
The payoff is that the Rust wrapper can read typed RecordBatch values through an Arrow StreamReader rather than parsing rows from an intermediate format like JSON and rebuilding typed Arrow columns. Field types, nullability and nested values are represented consistently on both sides. The boundary is not free — batches are still serialized, decoded and memory-managed — but the work stays batch-oriented and avoids per-record conversion, which suits a high-throughput delivery pipeline while still validating schemas against the Logpush dataset definition.
Schemas are not inferred from batch contents. Bufstream supplies a generated schema from each Logpush dataset's specification, giving DataFusion a stable view of available fields and types for query validation; the table named in the FROM clause selects which dataset schema a query uses.
SELECT ClientIP, EdgeResponseStatus, RayID
FROM http_requests
WHERE EdgeResponseStatus >= 500Planning, failure behavior and delivery
Saving a Transformer through the API or Transformer Studio stores its SQL in R2. Bufstream pulls the query through a multi-layer cache and translates it into a reusable DataFusion logical plan describing the scan, filters, projections and expressions. The plan is prepared once per time slice of a job's data and reused for every Arrow batch in that slice; for each new batch the Rust wrapper binds the batch as the plan's table input, executes the query and returns Arrow batches with the selected records and fields.
Failure handling is deliberately fail-closed. When a transform cannot run, Bufstream neither advances the job's delivery offset nor sends the untransformed batch, because shipping raw data in place of transformed data could break the schema, filtering or redaction assumptions of downstream pipelines. Instead it re-attempts failed transforms, with retries progressively delayed and errors reported on the job so a persistently failing job cannot starve healthy ones; jobs that stay failing for an extended period are disabled automatically. Transformer Health, in the Health tab of Logpush, surfaces successful and failed transformations, data volume before and after transformation, and transformation duration.
On the way out, Bufstream converts Arrow output back into NDJSON, batches and gzip-compresses it, and uploads to the configured destination. Destinations accept data at different rates and can temporarily slow down, so Bufstream adjusts the number of upload goroutines within each destination's concurrency limits.
Availability and setup
Transformers is available on Free, Pro, Business and Enterprise plans. Setup happens in Transformer Studio on the Logpush page of the Cloudflare dashboard: pick the dataset to transform, write the SQL query and preview it against a sample record, save the Transformer, then attach it to a matching Logpush job.
