Shaving log volume before it leaves Cloudflare
Cloudflare's Logpush product streams product logs to customer-owned destinations such as R2, Splunk, and AWS S3. The volume of that stream is often enormous: one representative customer receives roughly 21 billion gzip-compressed records per day, about 2.1 TB on the wire and close to 24.9 TB uncompressed. Storing all of it, even for short retention windows, forces customers to build and pay for ingestion pipelines that mostly handle data nobody queries.
Logpush jobs now support filters on the log fields themselves, so only records matching specified criteria are dispatched. An ecommerce operator, for example, might request only checkout-page HTTP logs where the bot score is non-zero; a security team might keep firewall events where the action was to block. Filters are configured either through the Logpush API or in the Cloudflare dashboard when creating or editing a job.
Alerting when a job stops delivering
A silent logging outage can be worse than no logs at all. Logpush now offers notification when a job has been retrying and failing to push for 24 continuous hours. Setup lives in the account-level notification settings: navigate to “Notifications” in the left panel, click “add”, and choose the Failing Logpush Job Disabled alert type.
Once saved, the alert delivers an email if the job is disabled after prolonged failure.
Health metrics for push jobs via GraphQL
To complement alerting, Logpush exposes per-job health statistics through the GraphQL API under the logpushHealthAdaptiveGroups node. The schema is self-documenting in any GraphQL client; the GraphQL analytics documentation covers the general query workflow. Available fields include records pushed, bytes pushed, compressed bytes, and the HTTP status of each delivery attempt.
The query below returns how many pushes to S3 ended with a status code other than 200:
query
{
viewer
{
zones(filter: { zoneTag: $zoneTag})
{
logpushHealthAdaptiveGroups(filter: {
datetime_gt:"2022-08-15T00:00:00Z",
destinationType:"s3",
status_neq:200
},
limit:10)
{
count,
dimensions {
jobId,
status,
destinationType
}
}
}
}
}This second example aggregates total records, bytes, and compressed bytes delivered:
query
{
viewer
{
zones(filter: { zoneTag: $zoneTag})
{
logpushHealthAdaptiveGroups(filter: {
datetime_gt:"2022-08-15T00:00:00Z",
destinationType:"s3",
status:200
},
limit:10)
{
sum {
bytes,
bytesCompressed,
records
}
}
}
}
}Operating Logpush at scale
Filters and health data make Logpush more practical to run as a primary log pipeline. Multiple jobs can point at different destinations, or different filtered subsets of logs can share a single destination, letting customers store only what their monitoring and compliance workflows actually consume. Customers without existing Logpush jobs can start by sending logs to R2.
