Automated Traffic in the First Half of 2020

Automated traffic now accounts for nearly 40% of all requests on the Internet, according to data from Cloudflare's network. The company processes an average of 18 million HTTP requests every second, giving it a broad vantage point for analyzing how bots behave across regions, time zones, and network types.

Cloudflare's bot classification system sorts traffic into four buckets: verified bots, definitely automated, likely automated, and likely human. Each request is assigned a bot score from 1 to 99, where lower scores indicate a higher probability of automation. Customers can use these scores in Firewall Rules, logs, and Workers to trigger actions, and Cloudflare's challenge platform—CAPTCHA and JavaScript challenges—offers a mitigation path that also feeds detection improvements.

The company relies on three detection mechanisms. A verified bot directory recognizes known good bots like search engine crawlers. A heuristics engine identifies non-verified bots via a managed directory. A machine learning engine marks requests with a high probability of automation. Because heuristic and ML detections are complementary, Cloudflare combines them when discussing "automated traffic" in aggregate.

Global Bot Behavior

Data collected between January 15 and July 31, 2020 shows that 60.6% of traffic on Cloudflare's network is likely human, while 19.3% is likely automated, 18.1% is definitely automated, and 2.1% comes from verified bots. Combined, 39.5% of scored requests originate from some form of bot.

Bot traffic follows a different rhythm than human traffic. Human requests fluctuate predictably with business hours, dropping 15% between midnight and 05:00 UTC and peaking at 25% above average from 14:00 to 17:00 UTC. Bot traffic is steadier, with smaller peaks and troughs. Verified bots are the most consistent of all, showing minimal hourly variance. This suggests that many bots—both good and bad—run the same tasks continuously with little human intervention, while some, such as sneaker bots, time their activity around product launches or business schedules.

A similar pattern appears across the week. Overall traffic dips about 4% on weekends, but verified bots only see a 1% drop, indicating they maintain a steady schedule regardless of the day.

Regional Distribution

Automated traffic is not evenly distributed around the world. North America is the dominant source, contributing over 50% of definitely automated traffic and nearly 80% of verified bot traffic. Europe is the second largest source, followed by Asia. Within North America, the United States, Canada, and Mexico account for 98% of all requests from the region. The United States alone generates 88% of total North American requests, 96% of its definitely automated traffic, and 98% of its verified bot traffic.

This is not purely a function of population. The three countries are home to roughly 497 million people, with the U.S. representing 66.5% of that total. Yet the U.S. still produces a disproportionate share of automated requests. Over half of all U.S. traffic is classified as automated, a marked departure from Canada and Mexico, where human traffic is more prevalent.

Networks Behind the Bots

Examining autonomous system numbers (ASNs) in the U.S. reveals where automated traffic originates. Residential ISPs like Comcast and AT&T dominate overall eyeball traffic, but cloud providers play an outsized role in bot traffic. Amazon and Google alone account for 11.5% of all U.S. eyeball traffic. For verified bots, the top ASNs are almost entirely cloud providers: Amazon, Google, Microsoft, Advanced Hosting, and Wowrack. For likely and definitely automated traffic, Amazon, Google, and Microsoft together represent about 30% of requests, with Comcast still contributing a notable 4.8%—evidence that some bots operate from residential networks.

COVID-19 Effects in Europe

The pandemic reshaped traffic patterns globally. Total traffic has grown 10% on average worldwide since the start of the year, but North America saw increases exceeding 40%. Some regions, like Africa and Asia, changed little. Europe experienced a surge that has largely normalized.

That normalization varied by country. Italy, among the first European nations to impose lockdowns, saw traffic rise weeks before its neighbors and eventually settle in line with the European average. Portugal also stabilized, but with busier weekdays. The United Kingdom, which delayed lockdown measures, saw traffic continue climbing past 40% above January levels without leveling off.

Definitely automated traffic showed less dramatic movement. Italy remained steady, Portugal saw a notable increase, and the U.K. tripled its baseline count. Verified bot traffic was also generally stable except in Italy, where it spiked sharply between March and May. This surge was not confined to a few high-traffic zones; it appeared across the top 10,000 zones. The cause was a single user agent: Google Image Proxy, which anonymizes image requests from Gmail. As users opened emails containing images, those requests flowed through Google data centers. The spike reflected a data center shift rather than a change in user behavior—when traffic was re-examined across multiple countries, the global average for this bot barely moved.

A Persistent Automated Undercurrent

The first half of 2020 saw web traffic climb roughly 35% as the pandemic pushed more activity online. Within that growth, automated traffic remained a constant presence. North America, led by the United States, is the primary source of bots, and cloud infrastructure is their primary launchpad. The tools for managing these requests—bot scores, challenge platforms, and behavioral analysis—give site operators the ability to filter the noise, provided they know what they are looking at.