A closer look at early TCP terminations

Cloudflare handles more than 60 million HTTP requests per second worldwide, with roughly 70% arriving over TCP connections. Not every new TCP connection results in a useful exchange, however: globally, about 20% of new connections to Cloudflare’s servers time out or are closed with a TCP reset before completing a request. This article examines those cases — connections that terminate within the first 10 ingress packets, which we call anomalous — and introduces a new Cloudflare Radar dashboard and API endpoint for tracking them in near real time.

The data builds on Cloudflare’s prior work on connection tampering, documented in a peer-reviewed study. Analyzing anomalous connections can reveal scanning activity, middlebox interference, denial-of-service attempts, connectivity problems, and other network behavior.

Normal versus anomalous TCP behavior

TCP connections progress through distinct phases: establishment, data transfer, and closure. Establishment uses the well-known three-way handshake: the client sends a SYN, the server replies with SYN+ACK, and the client completes the handshake with an ACK. Data transfer typically begins when the client sets the PSH flag on the first data packet, prompting the server to deliver data to the application immediately.

Closure happens in one of two ways, per RFC 9293:

  • Graceful close via FIN: Either party sends a FIN to indicate it has no more data. The other party acknowledges, and each direction closes independently.
  • Abort via RST: A party sends a reset, instructing the other to discard all connection state immediately. Resets usually follow an unrecoverable error.

Connections can also end by timeout when no data or acknowledgements arrive for a prolonged period. The RFC 9293 default is five minutes unless overridden by keepalive messages or application settings.

1622-2

We classify TCP connections as anomalous when they close via a client-side reset or timeout, especially before any meaningful data exchange occurs. Anomalous connections may be benign on their own but can signal broader issues, particularly in the early stages. Common sources include:

  • Scanners: Port scanners send SYNs to test for responsive servers but often fail to clean up connections once the probe gets a response.
  • Sudden application shutdowns: Browsers may send RSTs when closing tabs, or devices may lose power or connectivity, causing timeouts.
  • Network errors: Unstable links, such as severed cables, can lead to connection timeouts.
  • Attacks: SYN flood (half-open) attacks send repeated SYNs to exhaust server resources maintaining incomplete connections.
  • Tampering: Intercepting middleboxes may drop packets (causing timeouts) or inject packets to disrupt connection state — possible because TCP is unauthenticated and unencrypted.

Reading the dataset

The new Radar dataset classifies anomalous connections by the stage at which they terminated. The stage provides clues about the cause:

  • Post-SYN (mid-handshake): Reset or timeout after the server receives the client’s SYN and replies, but no ACK returns. IP spoofing is common here, so geolocation is unreliable.
  • Post-ACK (immediately post-handshake): Reset or timeout after handshake completion, before any data reaches the server.
  • Post-PSH (after first data packet): Reset or timeout after the server receives a packet with the PSH flag, such as a TLS Client Hello.
  • Later (after multiple data packets): Reset within the first 10 client packets, after several data packets have arrived.
  • None: All other connections.

The dataset is filtered by Cloudflare’s attack mitigation systems to keep focus on legitimate traffic. All data is passive, reporting observable events rather than root causes.

The global view shows that roughly 20% of new TCP connections to Cloudflare terminate anomalously within the first 10 packets, consistent with prior studies. Rates vary widely by country and network, which the global average obscures.

2544-2

Comparing countries and networks

In the United States, anomalous connection rates run slightly below the global average, mainly due to lower rates in the Post-ACK and Post-PSH stages — phases more indicative of middlebox tampering. Elevated Post-SYN rates appear in most networks due to scanning and potential IP spoofing. Higher reset rates in the Later stage often reflect human actions, such as browsers closing tabs with RSTs.

1622-4

A typical residential US ISP such as AS22773 (Cox Communications) shows rates comparable to the country as a whole.

1622-5

Contrast that with AS15169 (Google LLC), which originates many Google crawlers and fetchers. That network shows notably lower reset rates in the Later stage, likely because its traffic is predominantly automated and not driven by human tab-closing actions.

1622-6

Indeed, Cloudflare’s bot detection system classifies over 99% of HTTP requests from AS15169 as automated. This kind of corroboration — comparing the new anomalous connection data with other Radar metrics — illustrates how the dataset can support longitudinal studies, validate known behaviors, or surface new targets for investigation across networks worldwide.

Corroborating single signals with external data

Passive measurements at Cloudflare's scale provide broad visibility, but alone they cannot explain why a connection closed at a particular stage. A reset or timeout may stem from scanning, filtering, middlebox behavior, or application behavior. Root-cause determination requires complementary sources. In practice, this dataset is most useful for quantifying the reach of phenomena already documented by other research, such as reports from OONI or Censored Planet.

Consider AS398324, which presents a troubling picture: over half of connections show as anomalous in the Post-SYN stage. In fact, this is CENSYS-ARIN-01, a network belonging to scanning company Censys. The Post-SYN anomalies are likely due to network-layer scanning — the scanner probing with a single SYN packet and never completing the handshake. High rates of Later-stage anomalies and an automated-traffic classification near 100% point to application-layer scanning.

1622-8

via Cloudflare Radar

Indeed, like AS15169, over 99% of requests from AS398324 are classified as automated.

1622-9

via Cloudflare Radar

Detection of reset injection

This dataset originated from a research effort into active connection tampering, alongside work on HTTPS interception. A well-documented tampering mechanism is reset injection, commonly associated with the Great Firewall of China (GFW). Middleboxes employ this technique when they inspect data portions of packets and detect a forbidden domain name, usually in the TLS Client Hello via the Server Name Indication (SNI) field. The middlebox injects a forged TCP Reset (RST) to either or both parties, causing connection aborts. In this dataset, reset injection typically manifests as Post-ACK, Post-PSH, or Later anomalies — although these categories are not unique to tampering.

China provides a natural case study. Connections geolocated to the country exhibit higher rates of Post-PSH anomalies than the worldwide average. However, the rates vary significantly across individual Chinese networks, likely due to traffic composition or differing implementations of the technique. In contrast, Post-SYN anomaly rates are consistently elevated across most major Chinese ASes, which may reflect scanning, spoofed SYN floods, or residual IP-blocking collateral effects.

1622-10

via Cloudflare Radar

AS4134 (CHINANET-BACKBONE) shows Post-PSH rates lower than other Chinese networks but still well above the worldwide average.

1622-11

via Cloudflare Radar

AS9808 (CHINAMOBILE-CN) and AS56046 (CMNET-Jiangsu-AP) both show double-digit percentages of Post-PSH anomalies.

1622-12

via Cloudflare Radar

1622-13

via Cloudflare Radar

Identifying targets for follow-up study

Beyond confirming known behaviors, the dataset also surfaces networks that stand out for potential investigation — sometimes producing anomalies without obvious explanations.

Unattributable ZMap port scanning

One unexplained pattern originates from UK clients: daily, during an intermittent 18-hour interval, over 10% of connections time out at the initial SYN stage.

1622-14

via Cloudflare Radar

Internal inspection traced nearly all Post-SYN anomalies to a ZMap scanner on AS396982 (GOOGLE-CLOUD-PLATFORM), performing what appears to be a full port scan of all IP address ranges. The client self-identifies in line with responsible ZMap practice, but its purpose does not indicate an explanation for the scanning. Similar levels of scan traffic are visible from the same AS with US-geolocated IP prefixes.

1622-15

via Cloudflare Radar

Zero-rating enforcement on mobile networks

Country-level anomaly rates also disclose interesting trends. In Mexico, Post-ACK and Post-PSH anomaly rates exceed the global average — particularly notable since there is no documented evidence of government-led content blocking or filtering. The profile resembles other regional networks, suggesting a technical implementation shared among providers.

1622-16

via Cloudflare Radar

Nearly half of connections from AS28403 (RadioMovil Dipsa, operating as Telcel) are reset or time out immediately after the TCP handshake completes at the Post-ACK stage. A plausible mechanism is zero-rating, where a mobile operator permits data access to selected destinations even after a subscriber's data cap is exceeded. The provider would block connections to non-zero-rated resources, while allowing others. A rising explanation invokes the provider discriminating on SNI, which is sent in a data packet right after the handshake: if the packet carrying the SNI is dropped, the server observes the SYN and ACK packets but little else — matching the Post-ACK anomaly signature.

1622-17

via Cloudflare Radar

Peru, sharing a similar pattern, shows even higher Post-ACK and Post-PSH anomaly rates.

1622-18

via Cloudflare Radar

Within specific networks, AS12252 (Claro Peru) displays Post-ACK anomalies similar to AS28403 in Mexico. Both are operated by América Móvil, raising the likelihood of consistent network policies and management across markets.

1622-19

via Cloudflare Radar

Distinctly, AS6147 (Telefónica Del Perú) shows high Post-PSH anomalies — a marker suggesting the network enforces similar policies through a different stateful layer technique.

1622-20

via Cloudflare Radar

A closer look: time series and country-level signals

Because the measurement is continuous and passive, it can reveal how network behavior changes over weeks or months. Examining TCP resets and timeouts across short windows and longer horizons shows distinct patterns tied to real-world events, and correlating the data with other signals makes the picture much clearer.

During exam-related Internet shutdowns in Syria in mid-2024, Radar showed spikes in the rate of Post-SYN anomalies. These anomalies coincided with a near-total drop in traffic, including SYN packets, during the shutdown periods. A second round of shutdowns beginning in late July produced similarly prominent spikes. In Iraq, the same exam-related shutdowns produced multiple Post-SYN spikes, though less pronounced than Syria's.

1622-21

Algeria took a different approach during exam periods. Instead of full shutdowns, the country appeared to selectively restrict specific connections. During those windows, Radar showed an increase in Post-ACK connection anomalies. That pattern is consistent with a middlebox dropping packets carrying forbidden content while allowing others — such as the initial SYN and ACK — to pass through untouched.

1622-24

These examples show that the dataset is most useful when combined with other indicators. It is also exposed via the Radar API, so anyone can investigate further. The detection techniques themselves are transferable to other networks and servers.

Building the dataset at scale

Constructing the TCP resets and timeouts dataset required solving problems specific to Cloudflare's global footprint. The methodology has three steps: sample a subset of connections arriving at client-facing servers, reconstruct those connections from captured packets, and match them against signatures of anomalous behavior. The sampling is entirely passive — it cannot decrypt traffic and sees only existing packets on the wire.

A novel element is that only one direction of the connection, client to server, needs to be observed. Each signature has two parts: a connection stage and a set of tags describing specific behaviors drawn from published research and internal observations. These design choices keep encrypted packets private and allow the approach to be replicated anywhere without access to the destination server.

Sampling connections without a capture pipeline

Running packet captures on every client-facing server would not scale, and it would require knowing exactly where and when to look. Instead, Cloudflare samples connections from all servers and logs them to a central location for offline analysis.

Existing packet logging pipelines logged individual packets, but a connection consists of many packets. To detect anomalies, enough packets per connection must be observed. The solution combines a flexible logging system built by the DDoS team with two carefully crafted iptables rules.

The first rule randomly selects and marks new connections for sampling. The rate is one in every 10,000 ingress TCP connections — a balance between coverage and pipeline load. The rules apply only after packets pass the DDoS mitigation system, and only new TCP connections are sampled, since connections can be long-lived:

-t mangle -A PREROUTING -p tcp --syn -m state 
--state NEW -m statistic --mode random 
--probability 0.0001 -m connlabel --label <label> 
--set -m comment --comment "Label a sample of ingress TCP connections"

The rule lives in the mangle table, in the PREROUTING chain for incoming packets. It matches TCP packets with the SYN flag set and no prior state for the connection (--state NEW), selects one in 10,000 via the statistic module (--mode random --probability 0.0001), and marks the connection with a label using connlabel.

The second rule logs subsequent packets in a marked connection, up to a maximum of 10. That number is not special either — it is generally enough to capture connection establishment, request packets, and resets on connections closing prematurely:

-t mangle -A PREROUTING -m connlabel --label 
<label> -m connbytes ! --connbytes 11 
--connbytes-dir original --connbytes-mode packets 
-j NFLOG --nflog-prefix "<logging flags>" -m 
comment --comment "Log the first 10 incoming packets of each sampled ingress connection"

Installed in the same chain, this rule matches only packets from labeled connections and only the first 10 packets per connection, using connbytes to track packet counts. Matched packets are sent to NFLOG and picked up by the logging system for central storage and offline processing.

Reconstructing connections from packet rows

Logged packets land in ClickHouse tables, with each packet in its own row. Reassembling them into connections requires a working definition of a connection. The standard network 5-tuple — protocol, source IP address, source port, destination IP address, destination port — is used with several adjustments:

  • Only the ingress half of a connection is sampled, so response packets are not seen. In most cases the server response can be inferred from server configuration, and the ingress packets alone suffice to identify anomalous behavior.
  • Queries run in 15-minute intervals, grouping packets that share the same 5-tuple within each window. Connections may be truncated at the end of an interval; for timeout analysis, incomplete flows whose latest packet falls within 10 seconds of the cutoff are excluded.
  • Resets and timeouts most often affect new connections, so only sequences starting with a SYN packet are considered. Existing long-lived connections are excluded.
  • Packet interarrival timestamps are not precise, so only the set of packets matters, not their order. TCP sequence numbers can sometimes determine ordering, but this does not significantly affect results.
  • A small fraction of connections with multiple SYN packets are filtered out to reduce noise.

Each incoming connection is placed into one of four stages based on where it closed: Post-SYN, Post-ACK, Post-PSH, or Later. The stage at which an anomalous connection terminates is a strong clue as to why the anomaly happened. The stage alone is what Radar displays today, but matching connections against more specific signatures can yield deeper insight in many cases.

Tags for more specific connection behaviors

While the stage-based grouping of connections relies only on TCP flags, additional packet fields can reveal more about what happened. Packet inter-arrival timing, exact flag combinations, IP identification, IP TTL, and TCP sequence and acknowledgement numbers all carry useful signal.

One concrete example: the ZMap scanner sets the IP identification field to 54321 and the TCP window size to 65535 in its SYN packets (source code). Packets arriving with those exact values strongly suggest ZMap activity. The zmap tag captures this case.

Tags can also encode known signatures of middlebox tampering. Active measurements (e.g., Weaver, Sommer, and Paxson) have shown that some middlebox deployments disrupt connections in consistent ways, for instance by setting an IP TTL that differs from other client packets or by sending both a RST and a RST+ACK. Details on those signatures appear in the connection tampering blog post and the peer-reviewed paper.

The following tags are currently defined. The list is hierarchical; some tags only make sense if a parent tag is also set. For example, fin applies only when reset is also set.

  • timeout: terminated due to a timeout
  • reset: terminated due to a reset (packet with RST flag set)
    • fin: at least one FIN packet alongside one or more RST packets
    • single_rst: terminated with a single RST packet
    • multiple_rsts: terminated with multiple RST packets
      • acknumsame: all acknowledgement numbers equal and non-zero
      • acknumsame0: all acknowledgement numbers zero
      • acknumdiff: acknowledgement numbers different and all non-zero
      • acknumdiff0: acknowledgement numbers different with one zero
    • single_rstack: terminated with a single RST+ACK packet
    • multiple_rstacks: terminated with multiple RST+ACK packets
    • rst_and_rstacks: terminated with a mix of RST and RST+ACK packets
  • zmap: SYN packet matches ZMap scanner characteristics

These tags are not yet exposed in the Radar dashboard or API, but exposing them is part of the roadmap.

Next steps

The current resets and timeouts dataset should be immediately useful to network operators, researchers, and anyone interested in anomalous behavior on the Internet. Several extensions are planned:

  • Expand the tag set and make tags visible in the API and dashboard.
  • Extend the analysis to connections from Cloudflare to customer origin servers.
  • Add QUIC support. QUIC currently carries over 30% of HTTP requests to Cloudflare worldwide.

For a deeper look at connection tampering, see the accompanying blog post and paper, or explore the dashboard and API directly. Questions and observations can be sent to [email protected].