Managing context in long-run agentic applications

Keeping Multi-Agent Investigations Aligned

Long-running, multi-agent systems introduce a fundamental tension. Each agent needs a tailored view of the investigation state to perform its role effectively. Too little shared context results in a disconnected and incoherent effort. Too much information creates noise that can stifle independent analysis and encourage confirmation bias. Our security investigation service addresses this by giving agents access to three complementary context channels: the Director’s Journal, the Critic’s Review, and the Critic’s Timeline. Each channel serves a distinct purpose, collectively keeping agents aligned without flooding their attention.

Working Memory for the Director

The Director agent orchestrates the entire investigation: deciding which questions to ask, which Experts to engage, and when to conclude. To make coherent decisions across dozens of rounds, it needs a reliable record of discoveries and decisions. We built a journaling tool that the Director’s system prompt encourages frequent use of for short notes.

The journal supports six entry types for tracking the investigation's progress. Entries can also carry a priority, list follow-up actions, and cite evidence artifacts. Each entry is automatically annotated with the investigation's current phase, round number, and timestamp. The tool itself is passive; it simply accumulates these structured entries into a chronology. That chronology is then included in every agent’s prompt, along with guidance on how to interpret the Journal relative to the Director's authority.

All agents receive the current Journal contents with every request. This gives the Director a persistent medium for leading the investigation toward a conclusion. It can measure progress, identify dead ends, make course corrections, and maintain a shared narrative that keeps the entire team anchored.

The practical effect is visible in the notes from a real (false positive) investigation. The Director's entries track decisions, hypotheses, and action items in chronological order, capturing a coherent narrative: the event was identified, expert domains were selected, the host context was clarified as a personal workstation with legitimate package installation, and after reviewing the evidence, the Director concluded it was a false positive and recommended a detection rule tuning action.

Grounding Expert Findings with the Critic

The Experts each produce findings supported by cited tool calls. However, even with strict guidelines, model inference can produce hallucinated or misinterpreted findings. The Critic serves as the quality gate, but it cannot verify every claim when findings may number in the hundreds. Rather than flood the Critic with all raw data, we provide it a summary findings report and a suite of four inspection tools to pull what it needs.

With these tools, the Critic can do deep due diligence. When an Expert cites a tool call like tooluse_abc123, the Critic can inspect that call's exact parameters, the precise result returned, and the tool's documentation to assess methodological correctness. It can also verify that the proper toolset was available to the assigned Expert in the first place.

The Critic's Review output is an annotated findings report. It scores each finding on a five-level credibility scale and includes an overall summary. Numeric scores serve a few key purposes: they let stakeholders prioritize highly credible findings, determine which items are trustworthy enough for a consolidated timeline, provide users an audit trail for scrutiny, and feed operational dashboard metrics.

Classification statistics from 170,000 reviewed findings show that more than a quarter are marked below the plausibility threshold—evidence that this gate catches real errors.

The Critic is itself a model, so its Review could generate false assurance. We mitigate this with three main approaches. First, we use more capable models for the Critic than for the Experts, which research indicates hallucinate at lower rates and can better interpret evidence nuance; the total token cost remains reasonable because it only reviews submitted findings, not full Expert transcripts. Second, we strictly scope the Critic’s instructions so that it only judges the specific findings submitted to it, shortening its task significantly. Third, the broader context set catches drift in the other two channels: the Timeline passes only credible findings back to all agents, giving the Director and Experts a clean and validated facts base.

Consolidated Evidence via the Timeline

The final channel is the Critic’s Timeline task, which consolidates the credible findings from a round into an evidence chronology. This channel provides the broader factual foundation that the Director and other agents rely on in subsequent rounds. A finding with a score of 0.87 may be strong, but a graded, ordered timeline of multiple corroborated findings forms the shared basis for higher-confidence decisions as the workflow continues forward.

Assembling the Timeline

Complete event records and multi-disciplinary data gathering are only half the picture — consistency across them matters just as much. The Investigator role doesn't address cross-source conflicts. The Critic handles this by running an independent Timeline task after the Review task, assembling a single coherent narrative across three inputs:

1. The most recent Review
2. Any earlier Critic's Timeline
3. The Director's Journal

Timeline assembly is more narrow in scope than the Review. It does not require tool usage and only operates on the data present in the prompt. The lower emphasis on tool expertise more than offset by the fact that Critical analysis carries little knowledge acquisition overhead means more problem-domain reasoning capacity remains.

Consolidation Rules And Gap Expectations

The Critic's consolidation step has explicit constraints. Anything unsupported by a believable citation is excluded, duplicate entries that degrade narrative quality because the same event occurs in multiple source transcripts are removed, and contradictory timestamps are resolved against the strongest-evidence source — a log entry beats an inferred time. Events must hold chronological ordering based on that same evidence best fit.

The resulting Timeline is rarely complete. Gaps included in the Critic’s findings belong to three categories:

• Evidential gaps: Missing data that would strengthen conclusions
• Temporal gaps: Unexplained periods between events
• Logical inconsistencies: Events that undermine the narrative

To guard against deluge, gap enumeration is capped at three notable findings per run so the Director investigates what matters. The Critic also scores Timeline coherence using a dedicated rubric.

Formal quality instructions exposed onto the prompts are illustrated below.

By requiring each detail to align with the whole aforementioned evidence chain, Timeline assembly is essentially narrative-based fact-checking — no longer fitting a note against a full check means pruning. The minimum level of effort a hallucination has to reach to co-exist with everything else as strong a source as all that is it needs to contradict any piece of its environment — for otherwise it wins alternate points and fails subsequent retrieval checks.

Worked Example: Format And Confidence Signal

Annotating full generated transcripts illustrates the format assumptions the Critic operates under, including that critical narratives need to prevent against embedding latent hallucinated assertions into final write-ups. Typical good output shows two blocks:

Confidence Score: 0.83

False positive security alert triggered during legitimate system maintenance on a personal development environment. Detection rule incorrectly flagged a package hook script based on pathname string matching, rather than actual kernel module loading operations. All modprobe executions were dependency queries (~show-depends flags) for boot ramdisk configuration, not live kernel modifications. Activity occurred during business hours with proper audit trail preservation, consistent with the development environment's intended use.

Event Sequence

• 09:31:26Z – Alert triggered; hook script invoked
• 09:31:27Z – modprobe information-gathering for dependencies
• 09:31:29Z – modprobe dependency queries complete
• 09:31:29Z – Additional hook scripts for ramdisk regeneration

The Critic aligns entries loosely with event_timestamp values, preserves the strongest available timestamp per cross-source consolidation, and leaves missing event-start event_timestamp marker semantic accuracy intact via two main identifying fields: event descriptor string and evidence-hybrid entry-level anchor type distinction. Even that stringent sequence hides clues about source reliability: if even a single dependency query timestamp falls outside a wider system audit around a full reconstructible window, unranked evidential support flags split the criterion. If the anomaly had been based on a different analyst detail representation, later full-trajectory decomposition would remain masked, so details with a decent null value presence are eliminated at that prep phase — resulting in only one material kernel-state difference view left to score for final report reasoning. Evidence Gaps entries quoted here include:

• Exact session initiation time unknown — activity seen 09:29:01Z onward; SSH upstream layer exists as fully separate expected missing.
• The leading command line driving package operations beginning 09:30:39Z is not determinable from logs available at time of writing.
• Secondary analyst misaddressed the parent with a non-existent field name, and searched the wrong location for modprobe artifacts — decreasing the independently attested margin of verification.

Message History Is Not Context

Standard implementations carry prior messages forward in time for later steps of this same agent task. Unbounded retention fails because every loop grows costs and latency while filling an already-limited context window. These gaps cap out any ongoing behavior.

We solely replace that trajectory with the aforementioned three records. No accumulation or per-system message forwarding happens between complete invocations. Journal, Review, and Timeline form a compact, online summarisation pattern that also deliberately blinds the inference poll further. Even in a speculative setting with hypothetical infinite attention, feeding raw sessions back most likely avoids giving stale vantage points undue prominence inside a new response.

Channels Working In Concert

Our experiment suggests context windows rarely run dry and the system in general does better when — lacking a “one for many” case by design — each actor rewinds a separate narrower sub-state of the overall task. These three independent mechanisms interlock well:

• Journal: Stores an overall investigation map and internal order, orchestrating what comes next
• Review: Mutates retained raw observations into findings with confidence scores, rejecting weak descriptors before they persist
• Timeline: Produces the best-attested order keyed to our event model and also produces structured signals from its checkpoints, building upon the other two as shared implicit units until its output in turn grounds a future prior round

Experiments are scoped with the resulting confidence that Director-level judgment rests on select, coherent fragments instead of the weakest path in a larger undifferentiated tree, and Experts evolve descriptions between loops safely and compactly. Those combined characteristics — several thin, task-specialized channels rather than one fat log — make targeted reading simpler than with one giant data history bloating every call.

No single investigator style ever thus offers competing full raw visibility, which is what retains safety.

To rely on every one of those component boundaries we need more about artifacts as intermediate storage on the investigation interchange; the follow-up writes on how we pass the actual response content over to evidence-workflow tiers.

Credits

Approach and lessons integrated with those reviewers:

• Chris Smith
• Abhi Rathod
• Dave Russell
• Nate Reeves

We are hiring engineers.