Designing Agentic Security Investigations
Slack’s Security Engineering team manages an event ingestion pipeline that processes billions of events daily from a wide range of sources. During on-call shifts, the core responsibility is reviewing the alerts that our detection systems generate. In this post, we describe how we built an AI-assisted investigation service to improve our efficiency and strengthen our security posture.
From a 300-Word Prompt to Structured Steps
In late May 2025, we built a prototype that was effectively a 300-word prompt with five sections: orientation, a manifest of available data sources, an investigation methodology, output formatting for a markdown report, and a response classification taxonomy. We exposed a subset of our data sources using a simple "stdio" mode MCP server and used a coding agent CLI as the execution environment.
Results were inconsistent. The model sometimes produced strong analyses, cross-referencing evidence from disparate sources. At other times it jumped to convenient or spurious conclusions without sufficient scrutiny. The limitation became clear: prompts are guidelines, not mechanisms for fine-grained control.
Our solution was to break the complex investigation methodology down into separate, single-purpose model invocations chained together by our application. Each stage now has a mandatory JSON-schema output format, which restricts the model's final response. This structured output approach isn't free — overly elaborate schemas can cause failures, and it remains subject to hallucination. However, it converts an instruction like "question your evidence" into a dedicated task with far more predictable behavior.
A Persona-Based Team
Our design was influenced by research on multi-persona prompting in single invocations, as described in papers from Stanford/OpenAI and Microsoft Research. We were drawn to this persona concept, but to retain control we implemented each persona as an independent model invocation. Security tabletop exercise conventions also shaped the architecture. The result is a team of agents, where each agent/task combination has its own schema and orchestration logic.
Three persona categories exist in our model: the Director, the subject-matter Experts, and a meta-expert Critic.
- The Director Agent is the Investigation Director. It advances the investigation end-to-end, forming questions that become the expert's prompt, using a journaling tool for planning, and reading the Critic's condensed feedback.
- Expert Agents produce findings from their own tools in response to the Director. We currently field four: Access (authentication, authorization, perimeter services), Cloud (infrastructure, compute, networking), Code (source code and configuration), and Threat (threat analysis and intelligence).
- The Critic Agent assesses the experts' findings against a defined rubric, annotating each finding with analysis and a credibility score. Its weakly adversarial position mitigates hallucinations and ambiguous interpretations.
Because each pairing is a separate model invocation, we vary model versions, prompts, and tools per role.
The Knowledge Pyramid and Investigation Phases
This separation yields what we call a knowledge pyramid.
Experts at the pyramid's base interrogate complex data sources with many tool calls, making their output token-intensive. The Critic reviews that output, inspecting the experts' claims and the underlying tool calls and results, which also adds significant token overhead. The Critic then builds an updated investigation timeline from only the most credible findings and passes that condensed narrative back to the Director. We can thus use low-, medium-, and high-cost models strategically for the expert, critic, and director roles respectively.
Work itself runs in configurable phases.
Each investigation begins in Discovery, where the Director broadcasts a plan to the entire expert team to inspect all data sources. A Director Decision step lets the Director decide whether to advance to another phase, with prompt advice on the reasoning to apply. In Trace, the Director picks a specific expert to question based on the discovery results, and we can flexibly swap model parameters per phase. Conclude starts only when the Director has gathered enough information for the final report.
Production Architecture
Our working prototype used a coding agent CLI as a harness, but production requires more: live observation of investigations, presentation of past findings, ad-hoc launches, and integration with our detection tools. The service has three components:
- Hub: Provides the service API, persistent storage, general CRUD operations, and a metrics endpoint to visualize activity, token usage, and cost.
- Workers: Pick up queued investigation tasks from the API and stream an event stream back through the hub. Scaling workers increases throughput.
- Dashboard: Allows staff to launch and watch investigations in real time from the event stream. It also exposes per-invocation model details, which is invaluable during debugging.
Emergent Findings
We saw surprising, emerging results from this system. An investigation into a suspicious command sequence revealed an otherwise unrelated credential exposure in the process ancestry.
A lightly edited version of the report summary shows the system's emergent behavior:
Investigation Report: Credential Exposure in Monitoring Workflow [ESCALATE]
Summary: While investigating [command sequence], the investigation uncovered a credential exposure elsewhere in the process ancestry chain.
Analysis
The investigation confirmed that the command execution on [TIMESTAMP] was part of a legitimate monitoring workflow using [diagnostic tool]. The process ancestry shows the expected execution chain. However, critical security concerns were identified:
- Credential Exposure: A credential was exposed in process command line parameters within the ancestry chain, creating significant security risk.
- Expert-Critic Contradiction: The expert incorrectly assessed credential handling as secure while the critic correctly identified exposed credentials, indicating analysis blind spots that require attention.
The finding is notable because the expert disclosed no issue: the Critic detected the gap while inspecting the expert's work. The Director then shifted focus to the credential exposure, mandated mitigation, and flagged what the expert missed. The credential issue was sent to the owning team for remediation.
Early Results from Agent-Driven Investigations
Slack is still in the early stages of moving security investigations onto AI agents, but the first results are already measurable. The web-based dashboard lets on-call engineers launch and monitor investigations in real time, with each case producing an interactive, verifiable report that documents how evidence was collected, interpreted, and judged. The shift in work pattern is notable: on-call shifts now revolve around supervising a team of investigation agents rather than manually pulling evidence.
The most striking difference from static detection rules has been the agents' tendency to make spontaneous discoveries that weren't explicitly prompted. In the example report, the investigation surfaced material findings on its own, and similar discoveries have occurred repeatedly in practice—spotting weaknesses in IAM policies, flagging problematic code, and identifying other issues that a fixed rule set would have missed.
Future posts in this series will cover several areas where the system is still being refined:
- Keeping agents aligned and oriented when multiple investigative personas are working in parallel
- Using artifacts as a lightweight communication channel between investigation participants
- Designing effective human-in-the-loop collaboration between engineers and agents
Acknowledgements
The work described here has benefited from contributions by Chris Smith, Abhi Rathod, Dave Russell, and Nate Reeves.
