How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines

When AI Agents Lack a Codebase Map

AI coding assistants only perform as well as their grasp of your code. Pointed at one of Meta’s large-scale data processing pipelines—four repositories, three languages, and more than 4,100 files—agents floundered, burning cycles on exploration rather than producing useful edits.

The fix came from building a "pre-compute engine": a swarm of 50+ specialized AI agents systematically read every file and distilled tribal knowledge into 59 concise context files. The impact was immediate:

• Navigational guides now cover 100% of code modules, up from 5%.
• 50+ "non-obvious patterns"—underlying design choices invisible at the code level—were documented for the first time.
• Preliminary tests on six tasks show ~40% fewer agent tool calls per task.
• Complex workflow guidance dropped from ~two days of research to ~30 minutes.

Since the knowledge layer is model-agnostic, it works across most leading AI models. The infrastructure is also self-sustaining: automated jobs run every few weeks to validate file paths, find coverage gaps, re-run quality critics, and fix stale references. The AI isn't just a consumer of this system—it's the engine running it.

Why Context Matters: A Config-as-Code Nightmare

The target pipeline embodies config-as-code: Python configurations, C++ services, and Hack automation scripts spread across multiple repositories and six subsystems—configuration registries, routing logic, DAG composition, validation rules, C++ code generation, and automation scripts—that all must stay synchronized.

Existing AI systems handled operational tasks well: scanning dashboards, matching historical incidents, suggesting mitigations. But extending them to development work failed because the model had no map. It couldn't know that two configuration modes use different field names for the same operation, where swapping them silently corrupts output, or that dozens of "deprecated" enum values must never be removed to preserve serialization compatibility.

Without that context, agents resorted to guessing and re-exploring, often compiling code that was subtly wrong.

The Approach: Structured Knowledge Extraction

The solution used a large-context-window model with phased task orchestration, deploying a coordinated agent workforce:

• 2 explorer agents scoped the codebase.
• 11 module analysts read every file, answering five targeted questions.
• 2 writers authored context files.
• 10+ critic agents ran three rounds of independent quality review.
• 4 fixers applied corrections.
• 8 upgraders refined the routing layer.
• 3 prompt testers validated 55+ queries across five personas.
• 4 gap-fillers covered remaining directories.
• 3 final critics ran integration tests.

Each module analyst answered five core questions:

1. What does this module configure?
2. What are the common modification patterns?
3. What non-obvious patterns cause build failures?
4. What are the cross-module dependencies?
5. What tribal knowledge is buried in code comments?

The fifth question yielded the deepest insights. Analysts uncovered 50+ non-obvious patterns, such as hidden intermediate naming conventions—one pipeline stage emits a temporary field name that a downstream stage renames, making the wrong reference fail silently during code generation—and append-only identifier rules where deleting a "deprecated" value tears backward compatibility. None of this had been documented before.

A Compass, Not an Encyclopedia

Each of the 59 context files follows the "compass, not encyclopedia" principle: a tight 25–35 lines (~1,000 tokens) in four sections:

1. Quick Commands: copy-paste operations.
2. Key Files: the 3–5 files you actually need.
3. Non-Obvious patterns.
4. See Also: cross-references.

No filler—every line has a purpose, and all context files together consume under 0.1% of a modern model's context window. Beyond individual files, the team built a cross-repo dependency index and data flow maps that turn "What depends on X?" from multi-file exploration (~6,000 tokens) into a single graph lookup (~200 tokens). That's crucial in config-as-code, where one field change ripples across six subsystems.

An orchestration layer sits on top, routing engineers to the right tool automatically. "Is the pipeline healthy?" scans dashboards and cross-references 85+ historical incident patterns. "Add a new data field" triggers configuration generation with multi-phase validation. Users describe a problem; the system chooses the approach.

Results and Quality Gates

Quality needed enforcement, not assumption. Three rounds of independent critic agents raised review scores from 3.65 to 4.20 (out of 5.0), and all referenced file paths were verified—zero hallucinations.

Because the context files have a shelf life, the system refreshes itself every few weeks, validating paths, spotting coverage gaps, and auto-fixing problems. As the team puts it, context that decays is worse than no context at all.

Debating the Conventional Wisdom

Recent academic research found AI-generated context files actually hurt agent performance on well-known open-source Python repositories like Django and matplotlib. That deserves attention, but it tests a different scenario: models already absorb those codebases during pretraining, so context files amount to redundant noise.

Proprietary config-as-code is the opposite—full of tribal knowledge absent from any model's training data. The approach also sidesteps the research's pitfalls with design choices that push back on the findings:

• Concision: ~1,000-token files, not encyclopedic dumps.
• Opt-in loading: context is fetched only when relevant, not always on.
• Quality gates: multi-round critic review plus automated self-upgrades.

The bottom line: without context, agents burn 15–25 tool calls exploring, miss naming patterns, and produce subtly incorrect code. The cost of omission is measurably higher than the cost of maintenance.

Applying the Playbook Elsewhere

The approach scales beyond Meta's pipeline to any team with a large, proprietary codebase:

1. Find the tribal knowledge gaps. Look for where AI agents fail hardest—usually undocumented domain conventions and cross-module dependencies.
2. Run the five-questions framework. For every module: what does it do, how do you modify it, what breaks, what depends on it, and what's undocumented?
3. Keep it to "compass" size. Aim for 25–35 lines. Navigational guidance beats exhaustive prose.
4. Gate on quality. Independent critic agents should score and improve the output before it ships.
5. Automate freshness. Stale context causes worse harm than no context. Build periodic validation and self-repair.

Meta is expanding the framework to more pipelines across its data infrastructure while exploring whether automated refresh can spot not just stale context but emergent patterns—new tribal knowledge forming in recent code reviews and commits. This approach converted undocumented institutional knowledge into structured context for AI, and its value compounds with every subsequent task.