Why Email Detection Explanations Matter
Email security teams are caught in a constant race. Attackers refine phishing, business email compromise (BEC), and other campaigns daily, and defenders must adapt just as quickly to keep inboxes safe. The sheer volume of threat signals processed every day feeds multiple AI and machine learning models, allowing detection rules to be created and deployed at high speed.
That speed creates its own problem, though: security teams often have little idea what exactly was blocked and why. The detection pipeline blends signals from multiple models examining content, headers, links, attachments, and sender reputation. When a rule fires, SOC analysts typically see only a cryptic rule name in the investigation tab rather than the specific combination of factors that triggered the alert.
Consider a rule like BEC.SentimentCM_BEC.SpoofedSender. Internally, this indicates that the email contained no unique links or attachments (a common BEC pattern), that sentiment analysis models flagged it as highly likely BEC, and that sender spoofing indicators (such as envelope_from anomalies) were found. Without that context, analysts are left to reverse-engineer the logic from an opaque label, wasting time or risking releasing a malicious email from quarantine.
Bringing in an AI Agent with Guardrails
Cloudy, the AI-powered agent already available in the Cloudflare One suite for explaining gateway policies, was a natural candidate for this job. Extending it to translate complex detection logic into clear, human-readable summaries required overcoming one major obstacle first.
Addressing the Hallucination Risk
Early testing exposed the classic LLM weakness: hallucination. The model frequently generated inaccurate information about messages. That is acceptable in some log-analysis contexts, but dangerous for email security. A false claim that a malicious message is clean could lead an analyst to release it from quarantine, creating a potential breach.
The root cause was the complexity of the inputs. Bulk email scanning runs messages through multiple ML algorithms examining body content, attachments, links, IP reputation, and more. That same complexity that makes manual explanations difficult also caused the first LLM implementation to be inconsistent.
Three Steps to Stabilize Outputs
To reduce hallucination risk while keeping inboxes secure, the team added several safeguards:
- RAG implementation: Cloudy was restricted to drawing only from the detection dataset corpus via retrieval-augmented generation (RAG), grounding assessments in actual detection data.
- Model context enhancement: Critical context about internal model names was added. For example, "Churchmouse" refers to a group of sentiment detection models, not a single algorithm. Without this, the model driftingly defined the term via the common idiom "poor as a church mouse" — historically interesting, entirely irrelevant to security.
Testing shows the result is far more stable explanations with minimal hallucinations. For instance, the detection SPAM.ASNReputation.IPReputation_Scuttle.Anomalous_HC now yields this summary:
"This rule flags email messages as spam if they come from a sender with poor Internet reputation, have been identified as suspicious by a blocklist, and have unusual email server setup, indicating potential malicious activity."
Beta Access and Next Steps
Cloudy email detection summaries are now entering beta with a select group of users. The primary focus is validating that the guardrails hold and no hallucinations slip through that could lead to security compromises. Only after quality has been verified will access be expanded to all customers.
In the meantime, organizations can use the free Retro Scan tool to apply predictive AI models to existing inbox messages, identifying threats that can be remediated directly. Cloudflare Email Security remains available, or teams can use these insights to implement controls with their preferred solution.
Organizations interested in assessing their exposure can request a phishing risk assessment.
