Why AI-written phishing still fails
Email remains the primary vector attackers use to compromise or extort organizations. Phishing attacks have stayed ubiquitous because email is so central to business communication, and as attacker tooling has evolved, so have the techniques for targeting users while evading security controls. The release of multiple AI large language models (LLMs) has set off a scramble to find new applications for generative AI — and one obvious use is crafting phishing content.
Phishing depends on the attacker appearing authentic. That authenticity takes two forms: visual and organizational. Visual authenticity relies on logos and imagery to establish trust, while organizational authenticity leverages business dynamics and social relationships. LLMs help with the latter. Attackers commonly use LLMs to translate and revise rough drafts into more superficially convincing messages. More sophisticated operations pair LLMs with personal data harvested from compromised accounts to write personalized, organizationally authentic emails.
WormGPT, for example, can take a poorly written email and rewrite it with better grammar, flow, and voice, producing fluent text that more easily passes as authentic. Discussion forums for threat actors encourage members to draft messages in their native language and let the LLM polish them.
One attack type that benefits from LLMs — and can carry devastating financial impact — is Business Email Compromise (BEC). In these attacks, malicious actors deceive victims into paying fraudulent invoices, and LLMs can make those messages sound more organizationally authentic. But the same technology can craft other phishing messages too.
Yet LLM-crafted messages still depend on the user taking an action, such as reading a fraudulent invoice or clicking a link — something that can't be spoofed so easily. And every AI-written email is still an email, carrying sender reputation, correspondence patterns, and metadata alongside its text. With the right mitigation strategy, LLM-enhanced attacks can be reliably stopped.
Despite the recent spotlight on ChatGPT, LLMs are not new, and Cloudflare has been training models to defend against LLM-enhanced attacks for years. Analyzing all components of an email — not just the prose — ensures protection against well-written messages, because machine learning systems trained on billions of messages are not deceived by nicely-worded text.
The limits of AI-generated attacks
The riskiest AI-generated attacks are personalized using data harvested in earlier compromise operations. Attackers collect this information during traditional account compromise, iterate through the process, and strike once they have enough material. The attacks are highly targeted and specific. AI's advantage is scale of operations, but mass data collection is required to create messages that accurately impersonate someone else.
AI-generated attacks may offer personalization and scalability advantages, but their effectiveness depends on having sufficient samples for authenticity. Traditional threat actors can achieve similar results through social engineering, just without the efficiency AI provides. Fundamental limitations of opportunity and timing apply to all attackers, regardless of technology.
Defending against these attacks calls for a multi-layer approach: employee awareness training, advanced threat detection systems combining AI and traditional techniques, and continuously updated security practices to handle both AI and traditional phishing.
AI-generated attacks also come with tradeoffs. The number of attacks an actor can successfully conduct is limited by the opportunities available to them and the data they hold to craft convincing messages. Without access and opportunity, attacks are unlikely to succeed.
Opportunity, not language, gates BEC attacks
BEC attacks worry organizations because they can drain significant funds. Since BEC is text-based, it might seem that LLMs would make the problem much worse — but the reality is different. The main obstacle is opportunity: a window when events align for an exploitable condition. An attacker might use breach data to identify a gap in a company's vendor payment schedule, for instance. Even with motive, means, and resources, a BEC attack without opportunity falls flat. Some threat actors attempt volumetric "cold calling" attacks, but the vast majority of these fail — consistent with BEC's reliance on social engineering.
As an analogy: if someone walked into your business and demanded $20,000 without context, no reasonable person would pay. A successful BEC attack must bypass that validation step, and LLMs offer little help there. They can generate convincingly authentic text, but they cannot establish a business relationship, manufacture invoices that match a company's authentic style, or compromise vendor accounts. The largest BEC payments result from invoice compromise as well as account compromise — the former being necessary to supply convincing fraudulent invoices to victims.
Analysis of these attacks, based on scrutiny of hundreds of millions of messages per month, shows that non-textual trends define BEC attacks. Data suggests the vast majority of BEC attacks use compromised accounts. Attackers with access to a compromised account can harvest data to craft authentic messages that pass most security checks because they come from a valid email address. In the last year, 80% of BEC attacks involving $10K or more used compromised accounts. Of those, 75% conducted thread hijacking and redirected the conversation to newly registered domains — consistent with observations that most "successful" attacks leverage a lookalike domain that was almost always recently registered. Additionally, 55% of these messages involving over $10K attempted to change ACH payment details.
How multiple signals catch AI-crafted BEC
The message text contains no grammatical errors and reads easily, yet sentiment models triggered on the combination of urgency and an invoice — a common attack pattern. Several other models also fired. The attacker pretended to be from PricewaterhouseCoopers, but the domain from which the email was sent did not match. That sending domain was also registered recently, signaling illegitimacy. Additionally, a social graph model built from each customer's communication patterns flagged that, given the account's fresh history, this message was not business as usual. Combined with sentiment outputs, the analysis engine classified the message as malicious and prevented the recipient from interacting with it.
Generative AI continues to evolve rapidly, and more discoveries lie ahead. While AI-created BEC attacks may increase the raw number of attacks in the wild, their success rate should not rise for organizations with robust security solutions and processes in place.
What attackers still rely on
Cloudflare's 2023 Phishing Report, published in August of that year, examined roughly 13 billion emails and blocked about 250 million malicious messages. Despite the rise of ChatGPT and other generative AI tools, the dominant attack vectors remained familiar: malicious links and malicious file attachments. AI can make a message more readable and convincing, but it does not help attackers hide the dangerous payloads embedded in those messages.
Cloudflare's email security models examine every link and attachment they encounter. Links are crawled and scored based on domain information, on-page elements, and branding. The crawlers also look for input fields, which can indicate a credential-harvesting page. For links hidden behind redirects or geographic restrictions, the crawlers use the Cloudflare network to bypass those barriers.
Attachment analysis takes a similar approach. Because some parts of a file can be easily forged while others cannot, the systems break attachments down into primitive components and check those components for anomalies. This method is more reliable than traditional sandboxes, which attackers have found ways to evade.
Reading the full email, not just the message
An email contains more than its visible body and subject line. Defenders treat emails as having mutable and immutable properties. The body text is mutable and easily faked; a sender IP address is mutable but harder to spoof. Properties like domain age and domain similarity to known brands are immutable and cannot be changed at all.
Example email content
The message a user sees is only a fraction of the data available. The email headers contain a wealth of information that is typically hidden from recipients. These headers show the results of DMARC, SPF, and DKIM checks, revealing whether the message was authorized by the purported sender and whether it was altered in transit. Models can also access the sender's client IP address to check reputation and compare the sending domain against the branding in the message.
Example email headers
Because the body and subject are such a small part of an email, Cloudflare's models take a holistic view when assessing safety. Some models focus on the body, such as analyzing sentiment, but the final risk assessment combines models that evaluate every component of the message. That information is then surfaced to security teams using the products.
How the detection engine fits together
This multi-model philosophy is embodied in Cloudflare's SPARSE engine. In the 2023 Forrester Wave for Enterprise Email Security, analysts described the approach as using "preemptive crawling" to discover phishing infrastructure as it is being built. SPARSE combines multiple machine learning models, including natural language modeling, sentiment and structural analysis, and trust graphs.[1]
SPARSE is continuously updated with messages observed across Cloudflare's network. Analyzing billions of messages a year lets the team spot trends early and feed them back into the models. One example: in late 2023, Cloudflare noticed a rise in QR code attacks. Attackers used techniques like shrinking images or pixel shifting to defeat OCR scanners while still allowing phone cameras to resolve the malicious link. By feeding those messages into the models, which also considered other characteristics of the emails, Cloudflare was able to create detections that stopped the campaigns before they reached inboxes.
This preemptive scanning approach is designed to adapt as attacker behavior shifts. LLMs are a tool attackers are using more frequently now, but future techniques will emerge, and the same methodology is meant to absorb those changes as well.
Staying ahead of the shifting threat landscape
Email phishing remains difficult to defend against because attackers keep finding new ways to deceive users. As new technologies become publicly accessible, the field will keep changing. The use of generative AI will evolve, but the underlying approach—combining multiple models trained on different email properties—is what keeps customers protected as those trends come and go.
