Rethinking Warehouse Access for an Agentic Era
Meta’s offline data warehouse serves analytics, ML, and AI use cases at massive scale, making security a top priority. Teams across the company both manage access and depend on that data for daily work, but as data volume and access patterns grow in complexity, traditional methods for managing permissions are becoming a bottleneck. With the rise of GenAI and agents, we are rethinking how to enhance both security and productivity, making agents an integral part of internal data products to streamline access while minimizing risk.
Why Traditional Access Control Is Straining
We previously scaled data access by organizing the warehouse into a strict hierarchy: tables at the leaf level, with pipelines producing them and dashboards consuming them. On-calls manage these assets, followed by teams and organizational hierarchies, with business needs modeled as roles via role-based access control. In this rules-driven era, the data infrastructure could be visualized as a graph where assets (tables, columns, dashboards) are nodes and activities (queries, pipelines) are edges. Access decisions were highly local—engineers discovered data through teammates or by reviewing code, and approvals were granted to closely related teams.
AI systems, however, process data across vastly different portions of this graph, making these human-driven, localized decisions less viable. The complexity of the system increases as humans and agents work more frequently across domains, and AI is a major driver of these complex access patterns. The solution, we believe, also lies in AI. We needed to evolve our approach to an agentic workflow for data access, adapting a system originally designed for humans to serve both humans and agents working collaboratively. This requires native integration into data products, a streamlined user experience, and strict guardrails like rule-based risk assessment to safeguard the agents.
A Multi-Agent Architecture for Access
Our solution is a multi-agent system. Data-user agents assist users in obtaining access, while data-owner agents help data owners manage that access. These two agents collaborate to streamline the process when both parties are involved, but we keep them separate to decompose the problem and allow each to focus on its specific task.
Neither the user nor the owner agents are monolithic entities. The data-user agent is composed of three specialized sub-agents, coordinated by a triage agent:
- Suggesting alternatives: When users hit restricted tables, this sub-agent synthesizes "tribal knowledge" at scale to suggest alternatives—like unrestricted or less-restrictive tables, rewriting queries to use only non-restricted columns, or pointing to curated analyses.
- Facilitating low-risk exploration: Users often need only a small portion of a table during the exploration phase. This sub-agent provides context-aware, task-specific access for that low-risk work.
- Obtaining formal access: This sub-agent crafts permission requests and negotiates with data-owner agents. While there is a human-in-the-loop for oversight now, we expect these sub-agents to gain autonomy over time.
The data-owner agent also has dedicated sub-agents. One handles security operations, acting as a junior engineer that follows the team’s standard operating procedure (SOP) to process incoming permission requests. The other proactively configures access rules for the team, evolving traditional role-mining by leveraging data semantics and content.
Adapting the Warehouse for LLM Agents
LLMs communicate through text, and the warehouse’s hierarchical structure conveniently maps to a nested folder view. In this representation, organizing units become folders, and leaf nodes like tables, dashboards, or policies become resources, giving agents a read-only, summarized view. The SOPs governing access—derived from rules, wikis, and past interactions—also become text-based resources that guide both user and owner agents.
Context management is another crucial aspect, and we differentiate three scenarios:
- Automatic context: The system is already aware of who is trying to access what, so the agent fetches the exact context needed when a user’s access is blocked.
- Static context: Users can explicitly expand the scope of their focus from an automatic context to a broader one.
- Dynamic context: Agents filter resources further by metadata, like data semantics, or via similarity search.
We also model user intention, or "business needs," in two ways: explicit and implicit. In explicit intention management, a user tells the system their current task by assuming an associated role that carries the context of their business needs—this captures standard intentions. However, not every need can be modeled this way. For implicit intention, the system infers intent from a user’s recent activities. For example, if a data engineer is woken up at midnight to fix a pipeline failure, any subsequent data access is likely related to resolving that issue.
Deep Dive: Partial Data Preview
A complete use case shows how these elements work together. In a typical workflow, a data user moves from discovery to exploration before full analysis. During exploration, they require some data exposure, and we can enable task-specific, context-aware access through an orchestrated agentic workflow with four key capabilities:
- Context: We analyze user activities and information to align business needs with data controls, providing granular, task-specific access.
- Granular query-level control: We analyze the query’s shape—like whether it uses aggregation or random sampling—to determine if it is safe.
- Data-access budget: This is a first line of defense, based on the typical amount of data each employee accesses, which refreshes daily.
- Rule-based risk management: This acts as a defense against attacks or malfunctions of the AI agents themselves.
The architecture works as follows: the data-user agent taps into user activity feeds (including diffs, tasks, posts, and dashboards) and user profiles to formulate intent based on activities, profiles, and query shapes. It then calls upon the data-owner agent, which analyzes the query to identify targeted resources and fetches their metadata, such as table summaries, column descriptions, and SOPs. The data-owner agent uses an LLM to generate both the decision and its reasoning, which is then checked by a rule-based output guardrail.
All decisions and logs are stored securely for future analysis. While components like query analyzers are well established, this is the first time we are using LLMs to build a multi-agent system between data users and owners. LLMs excel here because business needs are often context-specific and difficult to model analytically, while agents construct the dynamic end-to-end workflow. Rule-based risk guards the agents' boundaries, and we keep a strong focus on transparency and tracing throughout the process.
Evaluation and Feedback
Evaluation is critical to developing any agentic system. We assess accuracy, recall, and other metrics against a curated dataset of historical requests, pairing them with the users’ activities and profiles. This evaluation runs daily to catch regressions.
We also built a data flywheel where queries, agent processing traces, context, and final outputs are encrypted and stored. This provides a secure audit trail and a foundation for a data tool that lets data owners review decisions and offer direct feedback, which we use to update our evaluations and refine the overall process.
Remaining work on the road to agent-readiness
Becoming fully agent-ready requires significant ongoing effort. Several key areas still need attention:
- Agent collaboration: Increasingly, data access comes from agents acting on behalf of users rather than direct human querying. Supporting these scenarios efficiently will demand new patterns and infrastructure.
- Tool evolution: The data warehouse and surrounding tooling were designed for employees and services, not software agents. These systems must be adapted so that agents can consume them effectively.
- Evaluation and benchmarking: Continual development of robust evaluation methods and benchmarks is essential to measure progress and ensure the architecture remains on track as agent use cases mature.
