Why volumetric DDoS detection falls short

Cloudflare’s always-on DDoS protection runs inside every server across its global network, analyzing incoming traffic for signals tied to previously identified attacks. Fingerprints are dynamically created to flag malicious traffic, which is dropped when detected in high enough volume. This approach works well for known attack patterns, but—much like the human immune system—it struggles with novel threats that require manual tuning to recognize.

With Cloudflare protecting millions of Internet properties and serving over 60 million HTTP requests per second on average, spotting unmitigated attacks is a significant challenge. Attacks targeting smaller sites can be just a few thousand requests per second, which is a tiny needle in that haystack. To close this gap, Cloudflare built an anomaly detection pipeline that automatically identifies unmitigated or partially mitigated attacks, allowing defenses to be updated against emerging threats.

The problems with simple volume baselines

A naive approach to attack detection might model traffic volume per minute for a destination, note that it follows a normal distribution, and flag any observation that deviates by a certain number of standard deviations from the mean—a z-score. Trigger thresholds set at z-scores of 3, 4, or 5 would catch traffic spikes that look statistical unusual.

Real-world traffic, however, is rarely so uniform. User load naturally varies over the day—take a meal delivery service with big peaks around mealtimes and low overnight traffic, for instance. That data no longer fits a normal distribution, pushing the threshold further away and letting smaller attacks slip through undetected.

This is especially dangerous for websites that elastically scale their hardware based on anticipated load. An attack that would be trivial to absorb during peak hours—when more servers are running—can overwhelm a site during off-peak hours when infrastructure has been scaled down to save costs. Even a few hundred requests per minute can be enough to take such a site down early in the morning.

Conversely, benign traffic surges are common. An e-commerce site running a Black Friday sale, or a site that gets 1,000,000 visitors from Reddit, would see a legitimate spike that the model wouldn’t anticipate, potentially causing real shoppers to be flagged as attackers. Choosing a sigma threshold that works across all customers is essentially impossible with this type of model.

Why time series forecasting doesn’t work either

Extending the volumetric approach with seasonality—assuming that yesterday’s traffic shape approximates today’s—leads to methods like Seasonal ARIMA (SARIMA). But those bring their own set of challenges.

First, capturing seasonality requires substantial data. Predicting weekly patterns needs at least a few weeks of history; predicting monthly or annual patterns (such as Black Friday) demands datasets over years. New customers would remain unprotected for a long time, which is impractical.

Second, the cost of retraining models is significant. Some time series models are only good for 2–3 inferences before needing retraining, which would mean retraining every model every 10–20 minutes across the network.

Third, and most critically, purely volumetric models break down when websites experience entirely benign spikes outside prior norms—flash sales, viral traffic, or Super Bowl commercials. No volume-based model can distinguish those from attacks.

Moving beyond volume: multidimensional anomaly detection

Volume isn’t the only axis on which traffic can be measured. Consider the proportion of end users across the top 5 browsers—over a given interval, that proportion should remain reasonably stationary, and it shouldn’t change dramatically during benign traffic surges. Through analysis, Cloudflare identified about a dozen such variables for each zone with three key properties:

  • They follow a normal distribution
  • They aren’t correlated, or are only loosely correlated with volume
  • They deviate from normal during “under attack” events

Expanding the z-score concept into multiple dimensions, each feature becomes a time series, and each point in time represents a point in the n-dimensional space. A multidimensional z-score would be the distance from the center of the point cloud, and a cutoff threshold could define an attack.

In practice, that point cloud won’t be a perfect sphere. Features measure different things on very different scales—one might vary between 100–300 while another stays in the 0–1 range. A shift of 3 in the latter would be significant, while in the former it would be unremarkable. Additionally, axes can correlate, so an increase in one dimension is mirrored by a proportional change in another, turning the sphere into a flat, off-axis disc.

Two mathematical techniques solve this. First, scale normalization: in each dimension, subtract the mean and divide by the standard deviation, centering the data around zero. That addresses scale differences but not the disc shape. Second, Principal Component Analysis (PCA) is used to determine the orientation and dimensions of the disc. PCA reorients the axes along the lines of correlation and rescales them to produce a well-behaved, n-dimensional sphere.

This process defines a coordinate transformation that maps any raw measurement to a location in the sphere. Since all dimensions are uniform after transformation, an anomaly score can be generated purely from distance from the center. A final scaling operation ensures spheres for different datasets are the same size, allowing a single, global cutoff distance λ across all models rather than per-customer tuning.

This metric is known as Mahalanobis distance. Dimensions with very little variance are discarded—if the disc is too thin, thickness isn’t useful, as such dimensions proved too sensitive in practice.

Training ~1 million models daily

This approach is highly parallelizable. A single model only needs traffic data for one website, and the datasets are small: 4 weeks of training data chunked into 5-minute intervals, roughly 8,000 rows per website.

Training and inference run in an Apache Airflow deployment in Kubernetes, which scales horizontally as needed. The system trains about 3 models per second per thread. Models are retrained daily, though little intraday model drift has been observed—yesterday’s model looks essentially the same as today’s—so training frequency might eventually be reduced.

Not every customer gets a model. Instead, Cloudflare trains models for a large representative sample, including many on the Free plan. The goal is to identify unmitigated attacks for further study, which then informs tuning of the existing always-on DDoS systems that protect all customers. This way, even the smallest sites benefit from defenses built on insights gained from detecting attacks against a subset of the network.