Cloudflare Log Explorer reaches general availability

Cloudflare has announced the general availability of Log Explorer, an add-on product that brings observability and forensics capabilities directly into the Cloudflare dashboard. The service builds on Cloudflare's network infrastructure to provide a contextualized view of logs across security and application performance events, with the goal of reducing reliance on third-party log analytics and SIEM tools.

The product has been in preview with several hundred customers over the past year. Early users report faster firewall rule tuning and easier log analysis for development teams compared to querying logs through a SIEM. Cloudflare cites several advantages of the native approach: lower cost and operational overhead from eliminating external data forwarding, faster detection through direct access to Cloudflare-native logs, full platform context during investigations, and the ability to move from investigation to mitigation without switching tools.

Custom dashboards for monitoring

Log Explorer includes custom dashboards that let users define metrics for monitoring unusual activity. Charts can be created using a natural language interface, similar to the AI Assistant introduced during Security Week 2024. The prompt translates descriptions into chart configurations that can be added to new or existing dashboards.

For example, a dashboard could monitor for Remote Code Execution (RCE) attacks detected by Cloudflare WAF, correlating them with other security events like malicious content uploads, source IP addresses, and JA3/JA4 fingerprints. An analyst might also use a natural language prompt such as "Compare HTTP status code ranges over time" to build a chart for error monitoring.

Expert-curated dashboard templates are available to jumpstart monitoring, covering areas such as:

  • Bot monitoring – identify automated traffic
  • API Security – monitor data transfer and exceptions at API endpoints
  • API Performance – view timing data and error rates
  • Account Takeover – track login attempts and leaked credential usage
  • Performance Monitoring – identify slow hosts and paths, view TTFB metrics
  • Security Monitoring – track attack distribution and correlate DDoS traffic with origin response time

Log Search for investigation

Log Search provides a streamlined query experience with data type-aware filters and an optional SQL interface for more complex queries. Log searches are also accessible via a public API. Using context gathered from dashboard monitoring, analysts can pivot to Log Search to investigate issues such as whether an attacker used compromised IP addresses to access other internal systems through Zero Trust services.

Queries built in Log Search can be saved for repeated use and shared with other Log Explorer users within an account, enabling collaborative investigation.

Custom alerting and flexible retention

Custom alerting, coming soon, will allow users to configure alert policies directly from Log Search. After defining and testing a query, users will be able to set a schedule interval and alerting policy, with the query running automatically on the defined schedule. Notifications can be delivered via webhook or PagerDuty.

Example alerting use cases include tracking error rate for a custom hostname using a query that calculates error rate per time interval, or detecting malware such as Katz Stealer by monitoring DNS queries and HTTP requests from Cloudflare Secure Web Gateway logs to identify infected machines.

Flexible retention, also coming soon, will let customers set precise log storage durations to meet compliance and audit requirements. Log Explorer is built on Cloudflare's R2 storage tier, so historical logs can be queried as easily as current logs, with no need for separate archiving or hot/cold storage configurations.

Architecture: running at Cloudflare scale

Log Explorer consists of three main components: ingestors write logs from Cloudflare's data pipeline to R2, compactors optimize storage files for efficient querying, and queriers execute SQL queries by fetching, transforming, and aggregating matching logs from R2.

During ingestion, each batch of log records is written to an Apache Parquet file in R2. Parquet's columnar format embeds metadata such as per-column minimum and maximum values, enabling queriers to quickly locate the data needed to serve a query. Logs are stored per customer, similar to Cloudflare D1, with plans in Q3 2025 to offer customer-defined retention policies and data region selection.

The service uses the Delta Lake open table format to provide a database table abstraction atop R2 object storage. A Delta table pairs Parquet data files with a transaction log that records every addition, removal, or modification of data files. Transaction logs are checkpointed periodically to avoid reading the full table history on every query. The transaction log also includes per-column min/max statistics for each Parquet file, so Log Explorer only fetches files that can potentially satisfy a query, and then uses the min/max statistics within each file to determine which row groups to fetch.

SQL queries are processed using Apache DataFusion, a Rust-based query engine, along with delta-rs, a community-driven Rust implementation of the Delta Lake protocol.

One challenge involved storage fragmentation. With logs ingested from Cloudflare's network spanning more than 330 cities in over 125 countries, writing directly to R2 would fragment storage into many small files, making queries prohibitively expensive. Log Explorer addresses this with a three-part strategy: logs are batched in Cloudflare's internal Buftee system, commits for multiple batches are staged and "squashed" before being written to the transaction log, and a background compaction process merges batches into larger files.

The team found that stock Delta Lake compaction was unsuitable for their workload. The default algorithm sorts files in reverse order of size and bins them to a target size, but merging logs without regard to timestamps destroyed data locality, leaving queries for a specific time frame fetching hundreds or thousands of files from R2. A custom compaction algorithm was written to merge ingested batches in order of minimum log timestamp, leveraging the min/max statistics. This reduced overlaps between merged files by two orders of magnitude, cutting some large queries from over a minute to just a few seconds.

Log Explorer is available as an add-on for self-serve and Enterprise plans. Customers can sign up through the Cloudflare dashboard or contact their account manager; developer documentation is available in the Cloudflare Developers site.