The Residential Proxy Blind Spot
Security teams fighting online abuse keep running into the same wall: botnets that route through residential proxies. These attackers cycle through IP address space until their traffic is indistinguishable from regular users, bypassing country blocks, ASN blocks, and rate limiting. Cloudflare's Bot Management machine learning model v8 targets this problem directly, identifying residential proxy abuse without resorting to broad IP blocking that would catch legitimate users sharing those networks.
Why Residential Proxies Break Traditional Defenses
Attackers have always used proxies to hide their origin and distribute attack traffic. IP rotation lets them sidestep reputation-based defenses and rate limiting. In response, defenders leaned on signals like data center IP ranges and known open proxy lists. That worked until bot operators shifted to proxies operating on residential network address space.
Residential proxy providers market access to pools of tens of millions of IPs on residential and mobile networks worldwide. These IPs are typically acquired by partnering with free VPN services or embedding SDKs into browser extensions and mobile apps, giving providers a foothold on victim devices and their network connections. Subscribers authenticate to a proxy gateway using HTTPS or SOCKS5, select a target country or region, and choose between sticky sessions or rotating IPs per request. The provider routes traffic through active exit nodes it controls on residential networks across the world.
The sheer size and diversity of these IP pools undermines defenses rooted in IP reputation and rate limiting. Attackers can rotate indefinitely, shrinking the window in which detection systems can act. An effective defense must identify residential proxy bot traffic from single request features to stop attacks immediately, or track unique fingerprints from the browsing agent regardless of the IP source. Overly broad IP block-listing would, by definition, also block legitimate traffic originating from residences where at least one device acts as a proxy node.
How the Model Gets Trained
The detection model is a chain of coordinated modules. Training and validation datasets are fetched from Clickhouse storage, where feature values and traffic aggregates are calculated across Cloudflare's network. The datasets use high-confidence labels for training; validation relies on customer-reported missed attacks, known bot traffic sources like verified bots, and high-confidence detections from other bot management modules. Apache Airflow orchestrates the pipeline, defining training, validation, and reporting dependencies as directed acyclic graphs (DAGs). Introducing a new dataset can be as simple as writing the SQL query that filters the desired request subset.
The Catboost model is trained with tuned hyperparameters, then evaluated against the current production default. Certain validation datasets are split into smaller slivers called specializations to capture behavior patterns that larger traffic sources would mask. For example, with heuristics engine detections serving as ground truth, specializations are defined per large ASN and HTTP version so accuracy improvements are measured for common patterns like HTTP/1.1 and HTTP/2 as well as rarer ones. The training DAG produces detailed reports covering accuracy, score distribution, feature importance, and SHAP explainers for each validation dataset and specialization.
Before deployment, the model must pass a readiness checklist covering reproducibility across multiple non-overlapping time ranges, plus checks for:
- Irregular feature value distributions, such as missing or skewed values
- Overlaps between training and validation datasets and feature values
- Diversity and label balance across training data
- Accuracy changes across validation datasets ranked by importance
- Overfitting signals via feature importance and SHAP explainers
Once validated, the model goes to shadow mode, observing live traffic in log-only mode without affecting bot scores. If confidence holds, it moves to beta customer onboarding and finally into active mode under continuous real-world performance monitoring.
Features That Catch Proxy Abuse
The model's inferences rely on features computed from single request properties and from patterns spanning multiple requests. These fall into three groups:
- Global features: inter-request aggregates calculated at scale for fingerprint and traffic source types, such as an ASN, across the global network.
- High cardinality features: fine-grained inter-request aggregates capturing local patterns, such as those for an individual IP address.
- Single request features: properties extracted per request, such as user agent.
The Bot Management system, BLISS, handles fetching and computing these feature values, making them available on Cloudflare's servers for inference by active model versions.
Network latency is not enough
Because residential proxy traffic must travel through additional hops before reaching its target, requests routed this way tend to exhibit higher network latency than direct connections. That observation alone can separate proxied from direct requests with a high true positive rate in a controlled lab environment. At Internet scale, however, it generates too many false positives. Users in regions that rely heavily on satellite Internet, for example, routinely show elevated latency for the majority of their requests due to the use of performance enhancing proxies.
Given the diversity of connection types across the Internet, network characteristics alone are an inadequate detection signal. The alternative is to look at how residential IPs behave. During periods of peak activity, IPs acting as residential proxies display a distinct pattern. Yet most residential proxy IPs only generate a small number of requests, and the temporary nature of proxy exit nodes creates significant periods of inactivity — when the browser or mobile app running the exit node is closed, the node disappears from the proxy network entirely.
One way to filter out those idle periods is to lengthen the monitoring window and penalize each IP that exhibits proxy-like behavior over time. That approach has a serious drawback: blocking by IP-based behavioral signals alone would catch legitimate users who unknowingly run apps or browser extensions that turn their devices into proxies. The problem is worse on mobile networks, where many users share IPs behind carrier-grade NATs.
Data from a 24-hour period shows that 4 out of 5 requests seen from active residential proxy IPs are actually direct, benign connections from residential devices. That finding pushed the detection strategy toward a per-request approach. By combining behavioral and latency features with new datasets, a fresh machine learning model can now flag residential proxy traffic while still allowing legitimate residential users on the same network to reach Cloudflare-protected sites.
Detection results from the field
Testing of the new model — v8 — began in shadow mode in March 2024. Every hour it classifies more than 17 million unique IPs involved in residential proxy attacks. Those IPs span more than 45 thousand ASNs across 237 countries and regions.
The most common endpoints requested through residential proxies are account takeover targets such as /login, /auth/login, and /api/login. When evaluated against previously missed attacks reported by customers, v8 shows substantial gains. In one case it correctly classified 95% of requests from distributed residential proxy attacks aimed at a customer's voucher redemption endpoint. In another, it caught a content scraping attack that had escaped earlier detection, with detection rates climbing during the traffic spikes shown below.
Cloud provider attacks also improve
Bot operators frequently run their scripts from cloud providers as well. The v8 training datasets now include improved ground truth labels for those attacks, and the early results are significant: v8 detects 20% more bots from cloud providers overall, and up to 70% more on zones marked as under attack. The list of cloud providers v8 recognizes is expected to expand in ongoing updates.
Existing Bot Management customers can enable automatic updates to their machine learning model to pick up v8's residential proxy detection now and receive future model improvements as they ship.
