How Meta understands data at scale

Building a shared language for sensitive data

Meta’s infrastructure spans a vast range of systems, each storing data in different shapes and formats. Understanding what that data actually means—its structure, context, and sensitivity—is a prerequisite for enforcing privacy protections. That need gave rise to the company’s Privacy Aware Infrastructure (PAI), a framework for embedding privacy tooling directly into Meta’s systems. Data understanding is the foundation of that effort.

The core problem is one of scale: millions of data assets, from structured tables to logs and AI models, are processed by millions of flows across Meta’s product family. To make sense of that volume, Meta built a data understanding pipeline built on three activities:

• Schematization — capturing the structure of each data asset
• Annotation — describing the meaning and semantics of that data
• Inventorying — registering everything in OneCatalog, Meta’s system for discovering and enumerating data assets

This approach represents a deliberate shift left: instead of classifying data after the fact, Meta integrates schematization and annotation into the product development process itself. The result is a privacy-aware workflow from the start rather than an audit at the end.

From ad-hoc detection to developer-integrated tools

Meta’s attempts to understand its data did not begin with a centralized system. Early efforts relied on heuristics and classifiers to automatically detect semantic types in user-generated content. That approach worked at a small scale, but as the volume of assets grew into the millions, the limitations became clear.

Running classification outside of developer workflows had two major drawbacks. First, classifications arrived late, creating confusion and duplicated work for engineers who had already moved on. Second, the outputs were hard to consume and interpret, which limited their usefulness across teams. Waiting for a separate privacy process to catch up with development was neither efficient nor reliable.

A universal privacy taxonomy for consistent semantics

Building better tooling was only part of the solution. To make data understanding reusable across Meta’s diverse product groups, the company introduced a universal privacy taxonomy: a standardized semantic vocabulary shared by all privacy tooling. With a common definition of what constitutes personal data, location data, or other sensitive categories, teams can build compliance features without reinterpreting the rules each time.

That consistency delivers quality at scale. The taxonomy ensures that any developer, working in any product, can rely on the same annotations, reducing friction and enabling the reuse of compliance infrastructure. For the privacy and product teams alike, it turned data understanding from a one-off effort into a shared foundation.

Adopting this framework across Meta required a flexible and incremental rollout. Not every team works in the same language or builds on the same systems, so Meta designed the onboarding process to accommodate that technical diversity. The process is supported by continuous collaboration between privacy and product teams, which proved necessary to unlock the full value of data understanding at scale.

A decade of incremental understanding

None of this happened quickly. Meta began cataloging and classifying its data assets more than ten years ago. Over that period, the systems have scaled to classify millions of assets on a daily basis, supporting a broad set of privacy initiatives across Meta’s product groups.

Data understanding is not a one-time migration; it is embedded into every stage of product development. Classifications are continuously updated as data and products evolve, ensuring private information remains protected even as the ecosystem grows and changes. This ongoing process is what makes privacy-aware development—and the infrastructure to support it—possible at Meta's scale.

Mapping sensitive data across Meta's infrastructure

Consider a profile in Facebook Dating. A user opts in to sharing religious views to find compatible matches. That single choice produces structured data that flows through dozens of heterogeneous systems — web, mobile, chat, backend services, and data warehouses. Each hop is a potential point where purpose limitation requirements could be violated if the data isn't tracked precisely.

Meta's five-step approach to data understanding is designed to give an end-to-end view of how such sensitive assets are tracked and protected. The goal is to ensure that religious views — or any other data subject to purpose limitations — are only used for the stated purpose, no matter which system is processing them.

The flow from online systems into the warehouse is where risk accumulates. Measures are in place to prevent religious belief data from being used beyond its intended purpose across all of these environments.

A canonical schema as the starting point

Meta's privacy-aware infrastructure initiative (PAI) begins with schematization — defining the logical structure of data assets in a standard format called DataSchema. Independent of any system implementation, DataSchema captures field names, types, metadata, and policies. It's built on the Thrift Interface Description Language, making it compatible across Meta's systems and languages.

DataSchema describes over 100 million schemas across more than 100 data systems. That covers granular units like database tables, key-value stores, logging streams from distributed systems, processing pipelines, and AI models. A data asset in this model behaves like a class with annotated attributes.

The source of truth (SoT) for a dating profile schema reflects this structure, defining the names and types of each field and subfield:

  - user_id (uint)
  - name (string)
  - age (uint)
  - religious_views (enum)
  - photos (array<struct>):
     - url (url)
     - photo (blob)
     - caption (string)
     - uploaded_date (timestamp)
Dating profile DataSchema

From this canonical SoT schema, system-specific representations — known as sources of record (SoR) — are derived. These are optimized for developer experience and the particular implementation constraints of each environment.

Predicting metadata with a universal taxonomy

To locate user data like religious beliefs at scale, schemas alone aren't enough. PAI layers on a universal privacy taxonomy — a common semantic vocabulary for data privacy management that works across programming languages and technologies. Classification systems scan Meta's product family using taxonomy labels to identify data elements and ensure privacy commitments are honored.

Annotations in this system are represented by taxonomy facets and their values. An asset might involve an Actor.Employee with data classified as SemanticType.Email originating from DataOrigin.onsite. The SemanticType annotation is the standard facet for describing what data means — names, email addresses, phone numbers, dates, locations.

Annotations grow stale as data models evolve, and new assets often lack them entirely. PAI's classification system addresses this with machine learning models and heuristics that sample data, extract features, and infer annotation values. Efficient techniques like Bernoulli sampling scale the process to billions of data elements with low-latency classification.

Several components make this work:

• Scheduling component: manages which data assets to scan, adapting to each system's architecture by either pulling data via APIs or receiving pushed data.
• Scanning service: accumulates samples in memory, deserializes rows (JSON, for example) into fields and subfields, and extracts features via APIs available in C++, Python, and Hack. It captures even ephemeral data.
• Classification service: applies heuristic rules for deterministic cases like date formats and user IDs, plus machine learning models trained on labeled datasets through supervised learning and refined with unsupervised pattern detection. Ground truth calibration keeps accuracy in check.
• Lineage and propagation: integrates classification rules with high-confidence lineage signals so exact data copies across systems receive equivalent classification without manual re-annotation.

Combining predictions with developer input

The integration of metadata predictions and developer input produces a full picture of a data asset — both its structure (schema) and its meaning (annotation). Portable annotation APIs plug into developer workflows, ensuring consistent representation across all of Meta's systems, accurate data understanding for privacy safeguards, and efficient evidence of regulatory compliance.

Two components work together. Classifiers generate predictions to aid developers; when confidence exceeds a threshold, annotation assignment is automated. Developers manually refine and verify annotations to capture context correctly, with human oversight guaranteeing reliability.

- user_id (enum) 		→ SemanticType::id_userID
  - name (string) 			→ SemanticType::identity_name
  - age (uint) 			→ SemanticType::age
  - religious_views (enum) 	→ SemanticType::faithSpirituality
  - photos (array<struct>):
     - url (url) 			→ SemanticType::electronicID_uri_mediaURI_imageURL
     - photo (blob) 		→ SemanticType::media_image
     - caption (string) 		→ SemanticType::media_text_naturalLanguageText
     - uploaded_date (timestamp) → SemanticType::uploadedTime

To keep schemas complete, data understanding is built into the data model lifecycle. Code representing a new asset's schema is auto-generated when missing, so no asset is created without a proper schema. For the religious beliefs feature, that means defining structure with fields like Name, EmailAddress, and Religion, then annotating the asset with Actor::user() so its fields are readily identifiable for privacy protection.

final class DatingProfileSchema extends DataSchemaDefinition {

 <<__Override>>
 public function configure(ISchemaConfig $config): void {
   $config->metadataConfig()->description('Represents a dating profile);
   $config->annotationsConfig()->annotations(Actor::user());
 }

 <<__Override>>
 public function getFields(): dict<string, ISchemaField> {
   return dict[
     'Name' => StringField::create("name")
        ->annotations(SemanticType::identity_name())
        ->example('John Doe'),
     'Age' => StringInt::create('age')
        ->description(“The age of the user.”)
        ->annotations(SemanticType::age())
        ->example('24'),
     'ReligiousViews' => EnumStringField::create('religious_views')
        ->annotations(SemanticType::faithSpirituality())
        ->example('Atheist'),
   ];
 }
}

Warehouse-specific representations

The same data asset looks different in each environment. In the data warehouse, the schema is represented as a Dataset — a Python class capturing the asset's schema and metadata, providing a native API for data pipelines.

​​@hive_dataset(
	"dim_all_dating_users", // table name
	"dating", // namespace
	oncall="dating_analytics",
	description="This is the primary Dating user dimension table containing one row per Dating user per day along with their profile, visitation, and key usage information.",
	metadata=Metadata(Actor.User),
)
class dim_all_dating_users(DataSet):
ds: Varchar = Partition("datestamp")
userid: DatingUserID = Column("User id of the profile")
email: EmailAddress = Column("User's email address"),
	age: PersonAge = Column("User's stated age on date ds")
	religious_views: ReligionOptions = Column("User's provided religious views")

Warehouse schemas incorporate rich types, a privacy-aware type system that encodes privacy metadata directly into the type definition. Types like DatingUserID, EmailAddress, PersonAge, and ReligionOptions are integrated into the schema, enabling enforcement of data quality rules at the type level. This catches issues like joining columns with incompatible user ID types before code executes.

ReligionOptions = enum_from_items(
    "ReligionOptions",
    items=[
        EnumItem("Atheist", "Atheist"),
        EnumItem("Buddhist", "Buddhist"),
        EnumItem("Christian", "Christian"),
        EnumItem("Hindu", "Hindu"),
        EnumItem("Jewish", "Jewish"),
        EnumItem("Muslim", "Muslim"),
  ...
    ],
    annotations=(SemanticType.faithSpirituality,),
)

A central inventory for all assets

With schemas and annotations in place, Meta's OneCatalog discovers, registers, and enumerates every data asset across the company's apps. It defines a data system as a logical abstraction over resources that persist data for a common purpose, exhaustively examining environments to register all systems hosting data assets.

Eligible systems enumerate assets through the platform, grouped by "asset classes" based on shared patterns. OneCatalog provides three guarantees:

• Completeness: regular consistency checks between defined configuration and stored inventory ensure all assets are accounted for and current.
• Freshness: beyond scheduled pull-based enumeration, the system subscribes to changes, updating inventory in real time.
• Uniqueness of asset ID (XID): each asset gets a globally unique, human-readable identifier following the structure asset://[asset-class]/[asset-name], enabling coordination and shared-key information exchange across systems.

On top of this inventory, OneCatalog provides a unified UI consolidating all asset metadata — a single point of access to view and manage assets. In the dating profile scenario, that interface shows the warehouse table asset with its full metadata and relationships.

Privacy teams across Meta rely on OneCatalog's central inventory for compliance assurance. The completeness and freshness guarantees make it a trusted source of truth, so teams can efficiently identify and address risks like unsecured data or unauthorized access.

Keeping data understanding current

Maintaining high coverage and quality of schemas and annotations means measuring precision and recall for both predicted metadata and developer-provided annotations. That baseline guides where security and privacy controls need attention.

The maintenance strategy spans four areas:

• Shifting left on creation time: intuitive APIs let developers provide metadata when creating assets, ensuring schemas and annotations are applied consistently downstream.
• Detecting and fixing annotation gaps: prediction signals surface coverage and quality gaps so new systems and workflows remain covered.
• Collecting ground truth: subject matter experts establish a baseline for measuring automated systems and driving continuous improvement.
• Providing canonical consumption APIs: common compliance patterns like detecting user data are handled through standardized APIs, ensuring consistent metadata interpretation with low entry barriers.

With schema and annotations front and center, engineers can build compliance solutions end-to-end, track and protect user data, and ship new features at scale — without losing sight of what the data means or where it came from.

From schemas to safeguards

Returning to the example: developers working on Facebook Dating rely on familiar APIs to define and annotate new data. These tools let them specify how their data is structured, and a data classifier validates those semantic-type annotations when the data is scanned during testing.

That classifier’s accuracy is checked against ground truth from subject matter experts. A feedback loop between product teams and the PAI organization keeps the shared taxonomy current, so every label remains meaningful. The end result is that privacy controls across Meta are built on the same canonical, cataloged metadata, helping the company sustain user trust and meet its obligations even as requirements shift.

In practice, that means the Facebook Dating engineers:

• Define structure and meaning with the standard schema and annotation APIs.
• Rely on automated classification to confirm semantic tags.
• Use expert-curated ground truth to validate the classifier.
• Feed outcomes back into the taxonomy through collaboration with PAI teams.
• Apply privacy policies using consistent canonical metadata.

Lessons from rolling out data understanding

Reaching this point was not just an infrastructure project. It required thousands of engineers across the company and several years of sustained work, but the pattern that emerged offers a blueprint for similar efforts.

• Make everything canonical. A global catalog of unique identifiers for systems, asset classes, assets, and taxonomy labels separates data understanding from how that metadata is consumed. That separation lets compliance tooling be built against a stable foundation.
• Start incremental, stay flexible. Instead of asking every team to rewrite their stack, Meta built a platform that imports schemas from existing implementations, embedding structure into legacy untyped APIs. The goal is to meet developers in their normal workflow—code, configuration, or UI—and add value sooner.
• Classification needs collaboration. Once the platform existed, infrastructure and privacy engineers partnered with subject matter experts to solve the hardest labeling problems: user-generateed content, data buried inside blobs, and a governed taxonomy that scales across teams.
• Feedback loops matter as much as launches. Product teams who co-built the integration gave immediate signal when schemas were wrong or classification drifted. That closed loop turned misclassifications into improvements, not just bug reports.

Where data understanding goes next

Data understanding is now embedded in how Meta approaches privacy at scale. By combining typed data models, annotated data collection, and policy processing, the company has automated annotation on millions of fields, columns, and tables that are critical to commitments—removing a substantial manual burden.

Beyond its current uses, the same foundation opens fewer obvious doors:

• Stronger AI: content moderation and recommendations could draw on richer data understanding to improve accuracy.
• Smoother development: internal tools can surface clearer data context, cutting down confusion and rework.
• Automated classification everywhere: storing metadata at scale lets teams locate the assets they need without digging manually, while speeding product work.
• New product capabilities: consistent understanding of data enables experiences that are more personalized and responsive.

Taxonomies still need to evolve to meet coming compliance demands, and there are fresh schematization methods to explore. Even so, canonical metadata is proving to be the backbone of data understanding, opening paths for innovation inside Meta and setting an example other organizations can adapt.