From warehouse to wallet: How Meta built data logs
In February 2024, Meta began including data logs in its Download Your Information (DYI) tool. Data logs provide users with additional details about their activity on Facebook, such as information about content they’ve viewed. This data can be unique or provide supplementary context to information already available through a user’s profile, Access Your Information, Activity Log, or account downloads.
Data logs are the latest step in a long series of transparency and control features Meta has rolled out over the years:
- 2010: Users can retrieve a copy of their information through DYI.
- 2011: Activity Log lets users review actions taken on Facebook.
- 2014: The "Why Am I Seeing This Ad?" feature offers transparency into ad targeting.
- 2018: Access Your Information provides a curated view of personal data. Instagram and WhatsApp introduce their own download and request tools.
- 2019: Users can view and clear their off-Meta activity. Meta joins the Data Transfer Project.
- 2020: DYI expands to include more interaction data from Facebook and Instagram.
- 2021: Access Your Information gets improved navigation.
- 2023: Access features are consolidated in the Accounts Center.
- 2024: Data logs become available in DYI.
Why data logs require a new architecture
Meta's production systems rely on techniques like caching to handle billions of queries per second. Its data warehouse, powered by Hive, is built for a different purpose: supporting low volumes of large analytical queries. It cannot scale to the query rates needed for real-time, per-user data access.
Data logs bridge this gap by providing users with formatted versions of individual Hive rows, processed to be transparent and easy to understand. The challenge is in the retrieval: Hive tables are partitioned by date and time, so pulling all data for a specific user requires scanning every partition to find relevant rows. With over 3 billion monthly active users on Facebook, roughly 99.999999967% of rows in a given table are irrelevant to any single user request. Scanning entire tables once per DYI request would be infeasible and wasteful. Caching data logs in an online, indexed system was also dismissed due to the computational and storage costs of copying warehouse data.
The batching design
The solution Meta settled on amortizes the cost of expensive full table scans by batching individual user requests into a single scan. This system provides predictable performance characteristics for infrastructure teams, even though most rows considered during a given scan are still filtered out as irrelevant to the users in that batch.
Here’s how the system works:
- A scheduled job collects recent user requests for data logs over a short time period into a single batch.
- The batch is submitted to a system built on Meta’s Core Workflow Service (CWS), which provides guarantees for executing long-running tasks with predictable performance and reliability.
- User IDs in the batch are copied into a new Hive table.
- For each data logs table, a worker task fetches metadata describing how to correctly query the data.
- For each partition, a Dataswarm job executes an INNER JOIN between the requester ID table and the column identifying the owner of each row. Jobs are configured with security and privacy policies, including access control lists and protections built on Meta’s Privacy Aware Infrastructure.
The output of these jobs goes into an intermediate Hive table containing data logs for all users in the batch. This INNER JOIN is expensive, requiring full table scans across relevant partitions. The output is then processed with PySpark to split results into individual files per user per partition.
The result is a set of comma-delimited text files with unfiltered raw data logs. A post-processing step in Meta’s Hack language applies privacy rules and filters, then renders the raw data into meaningful HTML files. Once complete, the files are aggregated into a ZIP file and made available to the requester through DYI.
Lessons from building data logs at scale
Several engineering challenges shaped the final system design.
First, the team learned the importance of robust checkpointing for incremental progress and resilience. Processing everything in a single pass might reduce latency, but a single failure would waste all prior work. Jobs could time out, and full-table-scan queries could run out of memory partway through. The ability to resume work piecemeal increases resiliency and overall throughput.
Data correctness was another critical concern. While building the component that splits combined results into individual user files, an issue emerged that could have returned data to the wrong user. The root cause was a Spark concurrency bug that partitioned data incorrectly across parallel workers. To prevent this, the team built verification into the post-processing stage, ensuring that the user ID column matches the identifier for the user whose logs are being generated. Even if similar bugs occurred in core data processing infrastructure, incorrect data would be caught before being shown to users.
Finally, complex data workflows require advanced tooling for fast iteration. Meta built an experimentation platform that allows running modified versions of workflows to test changes, with the ability to execute phases independently. This proved useful when innocent-looking changes, such as altering which column is being fetched, led to complex failures in data fetching jobs. The team can now run a test job under a new configuration whenever making a change to a table fetcher.
As Meta continues to enhance its access tools, the data logs system represents a significant investment in making warehouse data accessible to users. The batching approach, while requiring full scans that mostly process irrelevant rows, is a necessary trade-off to provide this information. Data warehouses are common across industries, so the design principles and lessons learned here could be useful for other companies seeking to provide access to their warehouse data.
From raw logs to readable records
Turning backend log data into something a user can actually parse is a multi-stage effort inside Meta. The raw values stored in the company’s systems rarely map cleanly onto what a person did in an app, so a cross-functional group of access experts works with product teams to review each dataset before it is exposed.
The first constraint is privacy at the record level. A single dataset can reference more than one person, but that does not mean everyone referenced should have equal access to it. If you block another user on Facebook, for instance, the block event is not shared with the person you blocked. Likewise, when you view someone else’s profile, that view is logged and shown to you, but not to the profile owner. The review process is designed to enforce those boundaries so data is never distributed to the wrong party.
There is also a line drawn around Meta’s own security and integrity systems. The company files millions of NCMEC Cybertip reports per year, and exposing the underlying data signals used to detect apparent child-safety violations could give away the detection techniques themselves. Those datasets are kept out of user-facing access tools entirely.
A more mundane but time-consuming problem is language. Internal strings that name system components are often meaningless outside engineering. A Hack enum describing user interface elements, for example, would read as jargon to a typical user — and sometimes to employees in other teams. Each of those internal references gets replaced with a curated, human-readable label that product experts verify for accuracy and consistency.
Consistency also means collapsing duplicate values. Engineers iterating on a feature will often log the same action through different parts of the stack, each with its own naming convention. In one real case from the Manage Activity tool, the action of moving content to trash was stored three different ways: MOVE_TO_TRASH, StoryTrashPostMenuItem, and FBFeedMoveToTrashOption. Through cross-functional review, the access team consolidated those into a single column header (“Which option you interacted with”) and a single label (“Move to trash”).
Once the mapping is finalized, the transformations are implemented in code using the renderers described earlier in this series. The renderers read raw values and convert them before display: a numeric ID like 1786022095521328 becomes “John Doe,” integer enums such as 0, 1, and 2 become “Disabled,” “Active,” and “Hidden,” and string enums are stripped of internal jargon. The result is a record that reads cleanly, as in the following example:
