Turning Specialist Knowledge into Institutional Memory
Organizations across industries face a familiar problem: the most valuable specialist knowledge is rarely written down. Models, playbooks, and checklists capture some of it, but the way experts actually reason — how they weigh competing priorities, interpret ambiguous situations, and apply judgment — remains locked in people's heads. In compliance-heavy environments, this means the same questions resurface across hundreds of reviews, each requiring days of manual research, and inconsistent expert assessments create real risk.
At Meta, we've built an AI agent designed to change that dynamic for a specific compliance domain. Instead of simply retrieving documents on demand, the agent encodes how an organization's experts think — their positions, procedures, and decision criteria — into a structured knowledge system that anyone can access, audit, and build upon. The system combines a knowledge architecture that separates what the agent knows from how it reasons, and a self-improvement loop that turns expert feedback into verified, regression-tested updates without retraining the model.
This approach is designed to generalize beyond compliance. Any organization with deep specialist knowledge governed by text — finance, security, engineering — can apply the same pattern to make expertise durable and accessible.
The Four-Layer Architecture
General-purpose LLMs give a strong starting point, but they lack the institutional context needed for high-stakes decisions. They cannot distinguish between what the organization could do from general knowledge and what it should consider doing based on historic positions, company direction, and business context. Closing that gap requires structuring the organization's own knowledge and priorities explicitly for the model.
The layers are deeply interdependent. The knowledge system's file structure enables automated editing; the reasoning layer's explicit procedures make failures easy to attribute; the evaluation framework gates every proposed change; and the improvement loop feeds results back into both knowledge and reasoning. Removing any layer degrades the rest.
Distilling Documents into Explicit Knowledge
Large organizations accumulate thousands of documents as byproducts of expert work. It's tempting to treat these as organizational knowledge itself, but they contain only the raw material. The real expertise — how experts reason, what they prioritize, and how they resolve ambiguity — remains implicit. An agent that retrieves chunks at inference time must re-derive that reasoning on every run, which is slow, inconsistent, and error-prone.
The solution is to make implicit expertise explicit ahead of time. A long-running offline process reads source documents and distills them into structured knowledge files: curated statements of how the organization interprets its domain, with constraints, boundaries, and machine-actionable routing implications. Those files become something general LLMs and static document stores cannot offer: the basis for a feedback loop where the agent can learn from human expert corrections without any model retraining.
The industry is converging on this concept. Andrej Karpathy's LLM Wiki structures agent knowledge as navigable file graphs; Google's Open Knowledge Format aims to standardize that structure across agents. The shared principle is that knowledge should be pre-extracted, explicitly structured, and progressively revealed, not re-derived at inference. We've extended this into a strict taxonomy with hard requirements for citation fidelity and institutional consistency, organizing 200+ files into four kinds:
- Position files capture authoritative organizational stances — how the organization has decided to interpret a given question, with constraints and boundary conditions, plus routing information that tells the reasoning layer when each applies.
- Taxonomy and vocabulary files provide a single source of truth for the organization's domain terminology, from entity types to classification tiers, ensuring the agent speaks the same language as the business.
- Routing indexes map input characteristics deterministically to relevant positions and procedures, avoiding reliance on embedding similarity alone.
- Gateway files define threshold tests the agent must pass before entering an analytical domain, preventing misuse of specialized knowledge.
Each file declares dependencies and consumers in YAML frontmatter. This forms a bidirectional graph that makes it possible to trace the ripple effects of any update — essential when automated edits are part of the system's design.
Where Wiki Ends and Retrieval Begins
A key architectural decision is how to split knowledge between the curated wiki and standard retrieval-augmented generation (RAG). The dividing line is information density and anticipated usage frequency.
High-density sources the agent consults on nearly every turn — positions, decision frameworks, boundary examples, strategic interpretations — belong in the wiki in distilled form. The wiki encodes the organization's evolving reasoning, so these files need to stay current, and the structure supports easy versioning, validation, and updating.
Sparse but situationally relevant sources stay in a semantic or lexical retrieval pipeline: detailed reference material, product specifications, historical decisions, niche external content. Loading these into the wiki would bloat the knowledge base and dilute the model's attention on every run.
The result is that the agent's core reasoning is always anchored in the most current organizational knowledge, while it can still pull in supporting evidence when a specific scenario demands it.
Composable Recipes for Expert Reasoning
Knowledge alone does not capture expertise. Experts follow structured methodologies — a senior compliance reviewer does not free-associate; they work through a defined analytical progression. Capturing those steps in an executable form for an LLM is the second half of the system.
The system defines composable procedures called recipes. Where knowledge files are declarative statements of position, recipes are imperative: multi-step workflows that specify what to examine first, which knowledge files to load at each step, which decision procedures to apply, and what constitutes a complete analysis.
The critical design constraint is keeping the two layers strictly separate. Recipes reference knowledge files but contain no facts. Knowledge files state organizational positions but never tell the agent how to reason with them. This separation pays off in several ways:
- Adding a new position means adding a knowledge file and updating a routing index — no recipe changes.
- Fixing a flaw in reasoning methodology means editing only the recipe.
- Failures attribute cleanly: either the knowledge was wrong, or the procedure was.
Recipes compose into pipelines. A top-level routing recipe inspects the input and dispatches to downstream recipes, each responsible for one analytical stage. This supports progressive disclosure: instead of one monolithic prompt covering every scenario, each step loads only the instructions and knowledge relevant to that phase. The impact was significant — early versions using a single flat instruction file plus semantic search consumed roughly 80% more tokens per turn than the recipe-driven structure, which delivers a small, targeted subset of knowledge at each stage.
Human Oversight as a Core Feature
Experts remain firmly in charge of the system's output. The agent is designed to accelerate expert work, not override their authority or judgment.
Two mechanisms enforce this. Checkpoints are deliberate pauses where the agent shows intermediate reasoning for review — the expert can confirm the direction, adjust, or overturn it before the work proceeds. Escalations trigger automatically when the agent encounters genuine ambiguity: underspecified inputs, or evidence that supports more than one defensible interpretation. In those cases, the agent hands the question to the human rather than forcing a questionable resolution.
Both mechanisms serve triple duty:
- Quality control: errors are caught before they propagate deeper into the analysis.
- Training signal: every human correction, escalation, or redirect becomes labeled input for the self-improvement loop.
- Trust building: experts gain confidence by observing the agent's intermediate reasoning and its willingness to flag uncertainty, rather than just inspecting a final answer.
For consequential domains like compliance, financial review, or safety analysis, keeping a human in the loop is not a defensive measure but a foundational choice that makes the entire system viable. Domain experts at Meta now field fewer routine questions and spend more time on the novel and ambiguous judgments where their knowledge matters most — with that same expertise compounded permanently into the agent's knowledge base rather than vanishing after each individual interaction.
Keeping an Institutional Knowledge Base Correct at Scale
Once an organization has codified expert knowledge into structured files and recipes, a new problem emerges: how do you keep that system correct as experts refine their positions and the dependency graph grows? Manual maintenance of interdependent knowledge files quickly becomes impractical. A single expert correction can require weeks of work because someone must trace every cross-reference, check for breakage elsewhere, and validate the fix under realistic conditions.
RAG memory systems and model-weight knowledge editing have explored how agents store and retrieve information, but far less attention has gone to maintaining a document-based knowledge base as it evolves. Many agents can draft their own fixes, but few apply rigorous validation to structured knowledge without model retraining. We treat maintenance as a compilation problem and automate it. Every expert correction moves through four stages: diagnosis into actionable issues, compilation into minimal verified edits, validation for correctness and regressions, and human expert review. Completed fixes are added to the regression suite so future cycles must preserve them.
Diagnosing What Actually Went Wrong
Raw expert feedback arrives embedded in conversation traces. The diagnosis phase must turn that into structured questions about root cause. An early approach classified by conversational form: if an expert provided information, it must be a knowledge gap; if they redirected the agent, it must be a procedure problem. That heuristic failed. Correcting a conclusion could indicate a missing fact, a flawed reasoning recipe, or genuine ambiguity in the source material.
The working method separates extraction from classification. First, pull every substantive expert signal along with a full manifest of what the agent loaded and how it was used. Then read the actual knowledge files and apply one test: could the agent have reached the right answer from what it had?
- Materials contained the answer but the agent erred: recipe problem.
- Materials lacked the answer: knowledge gap.
- Experts themselves disagree: ambiguity, escalated for human discussion.
Compiling Minimal Edits with Adversarial Review
The compiler translates each diagnosis into surgical file edits. Sub-agents analyze impact in parallel, looking at cross-references, conflicts with existing positions, token budget effects, test coverage, and duplication risk. Two design choices make the output trustworthy. First, an independent adversarial reviewer runs in a fresh context with no knowledge of the improvement rationale. It sees only the proposed diffs and hunts for introduced contradictions, broken edge cases, or undermined positions. Because it shares no context with the proposing agents, it cannot inherit their blind spots. Second, deterministic structural validation runs as a linter: dangling cross-references, budget violations, identifier collisions, and dependency cycles all fail programmatically. This layer is binary — pass or fail — not probabilistic.
Proving Fixes Before They Land
Every proposed change passes through two-stage validation. Targeted replay runs the agent on the original scenario that prompted the feedback, without the agent knowing it is being tested. A separate judge — unaware of what changed — evaluates the output against the original expert feedback. If replay fails, compilation is retried.
Next comes regression testing: multiple benchmarks for the domain, usually structured Q&A suites. For analytical domains with potentially multiple correct answers, an independent LLM judge scores each case pass/fail against defined criteria. The agent runs in parallel sessions against benchmark questions; any performance regression triggers a compilation retry. The retry prompt includes where the agent regressed along with the original issue and attempted fix.
The pipeline emits a pull request with a full audit trail. A human expert reviews a proven fix rather than debugging a raw failure. Once approved and merged — updating the knowledge file or recipe — the original failing scenario and its validated answer join the regression suite automatically. Each fix permanently raises the standard, and future changes must preserve whatever behavior was just corrected.
Measured Results Across Six Weeks
Three development sprints produced several concrete gains. Domain subject-matter experts rated outputs useful almost all the time, a clear jump from early versions that frequently demanded substantial rework. Individual assessment time dropped from days to minutes. Validated knowledge edits arrived at a rate that previously consumed full engineering sprints. Across all improvement cycles, zero regressions occurred, with every fix strengthening the test suite. Experts consistently reported that the agent handled the vast majority of analytical work, freeing them for the genuinely ambiguous cases requiring human judgment.
Applying the Architecture Beyond This Domain
Our build targeted a workflow that synthesizes dozens of internal positions and external sources into risk-weighted assessments. The architecture applies wherever specialist knowledge exists as tribal knowledge, consistency across assessments matters, volume exceeds expert capacity, and general-purpose LLMs fall short of institutional quality. Regulatory compliance, protocol adherence, financial risk assessment, security review, engineering standards, and procurement evaluation are all natural fits. The common thread is the need for genuine domain expertise, not generic reasoning.
Adoption requires four elements: a structured knowledge system with explicit file boundaries, cross-references, and a dependency graph; a procedural layer that separates analytical methodology from domain facts; an automated evaluation suite that grows with every improvement cycle; and human-in-the-loop checkpoints calibrated to risk tolerance.
The underlying principle is to keep complexity in text files readable by humans and agents rather than in model weights. Every improvement becomes a text edit a domain expert can review in seconds. Every change is version-controlled, diffable, and reversible. The compilation pipeline may be intricate, but its outputs are always transparent.
The result is a system where expert effort compounds. Each interaction makes the system better; each correction persists as a verified improvement. Organizational knowledge stops being trapped in individual experts and becomes available consistently, at scale, to everyone who needs it.
