Cloudflare’s Magic Network Monitoring Reaches General Availability

Cloudflare has announced the general availability of Magic Network Monitoring, a cloud-based network flow monitoring product that provides end-to-end visibility into network traffic. Formerly known as Flow Based Monitoring, the service is now available to all enterprise customers, with automatic enablement for Magic Transit and Magic WAN subscriptions.

The tool collects network flow reports — streams of packets sharing the same source, destination, protocol, and ports — from customer routers or other flow generators. This data is sent to a public endpoint on Cloudflare’s anycast network, making it possible to analyze traffic that never traverses Cloudflare’s infrastructure. Once ingested, flow data is displayed in an analytics dashboard that tracks traffic volume in bits or packets over time, source and destination IPs, ports, protocols, and router IPs. Alerts can be configured to flag DDoS attacks or other anomalous traffic volume patterns.

DDoS detection and mitigation workflow

Magic Transit On Demand (MTOD) customers are expected to benefit significantly from the new service. Previously, many of these customers lacked visibility tools needed to confirm DDoS attacks quickly. With flow analysis integrated, MTOD customers can now receive alerts when their network flow data indicates an attack, then activate Magic Transit either manually or automatically to initiate mitigation.

For Magic WAN and Cloudflare One customers, the value lies in extending visibility beyond traffic that already passes through Cloudflare’s network. Local network traffic, or flows exchanged between cloud environments, have historically been blind spots for these customers. Adding Magic Network Monitoring to that suite allows for monitoring of the full range of network traffic rather than only the segment routed through Cloudflare.

Sampling in network flow data

Network flow collection relies on statistical sampling. A router or flow-capable device examines one out of every X packets, where X is a configured sampling rate. Cloudflare documentation notes typical rates from 1-in-1,000 to 1-in-4,000 packets, though the optimal rate depends on traffic volume, traffic mix, and hardware resources.

Sampled data is packaged into one of two formats:

  • NetFlow — sampled packets are grouped by characteristics such as source/destination IP, port, and protocol, with a traffic volume estimate included for each group.
  • sFlow — the full packet header is retained as the representative sample without summarization, yielding richer, more detailed traffic data.

Flow records in either format are forwarded to Magic Network Monitoring, which performs its own additional sampling to scale across Cloudflare’s infrastructure.

Accuracy improvements in traffic estimation

Magic Network Monitoring’s early access release had a known issue: customers reported that traffic volume estimates were too high and diverged from baseline expectations. The root cause was inherent in the data structure. Flow records arriving at Cloudflare had already been sampled on the customer’s router, and the additional sampling layer introduced compounding error.

The first implementation used simple random sampling, selecting a random subset of flow records per timestamp to represent traffic at that instant. This proved problematic because flow samples carry unequal significance. Each record includes a weight based on the volume it represents, and those weights tend to follow a long-tail distribution. Random selection occasionally picked an outlier from the tail end, producing false spikes in volume analytics.

VarOpt reservoir sampling

To address this, engineering replaced random sampling with VarOpt, a reservoir sampling method designed for data streams of unknown length. The implementation uses a fixed-size reservoir that begins empty. New flow samples fill it until capacity is reached; as additional data arrives, random old samples are discarded to make room for new ones. After a set number of samples have been observed, the aggregate traffic volume across all weighted samples in the reservoir is calculated, yielding the volume estimate for that interval. The reservoir is then emptied and the process restarts with the latest incoming samples.

The switch to VarOpt significantly reduced estimation errors and eliminated the phantom traffic spikes seen in early access. Cloudflare states that this improvement was central to the product’s readiness for general availability.

Available now

Enterprise customers with Magic Transit or Magic WAN already have access to Magic Network Monitoring and can begin configuration through the Cloudflare dashboard under “Analytics & Logs > Magic Monitoring.” A free version with traffic volume limits is available on request to the Cloudflare account team for enterprise customers wanting to evaluate the service. That free tier is also in closed beta for Free, Pro, and Business customers.

Documentation is available through Cloudflare’s Developer Docs, and a dedicated Magic Network Monitoring channel exists on the Cloudflare Discord server for configuration help and product feedback.