Security Event Alerts: Raising the Bar with Thresholds

Security Event Alerts notify you when requests matching your security rules spike abnormally. These spikes can indicate an attack, but distinguishing real threats from natural traffic variations requires a careful detection strategy. Our previous approach relied exclusively on z-score analysis. Now, we introducing a new threshold component, creating a hybrid approach that reduces noise while catching genuine threats.

Why Z-Scores Alone Fall Short

Z-scores measure how many standard deviations a data point sits from the mean. In our setup, a z-score above 3.5 triggers an alert, a value tuned through analysis of customer data to balance sensitivity against false alarms. This method works well for most domains, but breaks down when a property has long stretches of zero security events.

For domains with many zero values, the mean and standard deviation collapse toward zero. When a non-zero value finally appears, it is mathematically infinite standard deviations from the mean, guaranteeing an alert—even for trivially small spikes like five events that no one would consider a threat. The result: customers with low traffic were flooded with alerts for harmless activity.

Option 1: Ignoring Zero Values

One possible fix involve excluding zeros from the mean and standard deviation calculations. This approach solves the issue for the five-event spike, lowering the z-score to zero and suppressing the false alarm. However, it introduces a new problem: with only one non-zero data point, that single value defines both the mean and the standard deviation, so the z-score always equals zero.

Consider a jump to 1,000 security events—an obviously significant spike. With just one previous non-zero data point, the z-score remains at zero and never approaches the 3.5 alert threshold. The calculation would miss genuine threats entirely.

Option 2: The Threshold Solution

Neither including nor excluding zeros gives reliable results alone. The true solution combines z-scores with a static threshold representing the minimum number of security events required to trigger an alert.

This hybrid condition requires both criteria to be met:

  • The z-score must exceed 3.5 standard deviations from the mean
  • The raw number of security events must cross a threshold of 200

In the 1,000-event spike example mentioned above, the threshold catches what the z-score misses. A spike of this magnitude crosses the 200-event threshold, and with enough historical data points, the z-score also confirms the anomaly. Both conditions are fulfilled, and an alert fires.

The threshold is calibrated so that larger, noisier spikes are filtered out, while actual threats still trigger alerts. This value was selected through analysis of historical attack data on our customer base. The threshold works in tandem with z-scores, not as a replacement—it ensures that insignificant spikes never distort the overall picture, while the z-score validates that an event is a true outlier relative to the domain's baseline.

Availability and Setup

This combined z-score and threshold methodology is now active by default for all WAF customers, requiring no configuration changes. Enterprise accounts will see the threshold applied across each alert type enabled on their domains. As we continue refining the alerting system, future iterations will aim to tailor this algorithm even more closely to each Internet property's unique traffic profile.