Rebuilding Meta’s Data Ingestion Pipeline
Meta’s social graph relies on one of the world’s largest MySQL deployments. A data ingestion system incrementally scrapes several petabytes of that data each day into the warehouse, feeding analytics, reporting, and machine learning workflows. The legacy architecture, built around customer-owned pipelines, worked at smaller scales but became unstable as landing-time requirements tightened.
Meta has since migrated 100% of its ingestion workload to a new architecture: a self-managed warehouse service designed for hyperscale efficiency. The migration itself—spanning tens of thousands of jobs—demanded new strategies in lifecycle management, rollout controls, and automation.
Defining a Safe Migration Lifecycle
Every job had to pass through a structured lifecycle with strict promotion criteria before the old system could be switched off. Verification required:
- No data quality issues—row counts and checksums must match between old and new system outputs.
- No landing latency regression—the new system must match or improve on the old system's delivery speed.
- No resource utilization regression—compute and storage usage must be comparable or better.
- Extra agreed criteria for critical tables, defined with downstream teams.
Phase 1: Shadow Testing
Each job first ran as a shadow in pre-production. The shadow consumed the same production source data but wrote to an isolated shadow table. This exposed the new system to real workloads while keeping failures contained. Engineers continuously compared row counts and checksums between production and shadow outputs, investigating and fixing any mismatch before proceeding. Compute and storage requirements were also measured to confirm production capacity.
Phase 2: Reverse Shadow
Once stable, the roles flipped: the shadow job wrote to the production table while the original production job wrote to the shadow table. This arrangement preserved ongoing data-quality comparisons after rollout and enabled fast rollback without reconfiguring the old job—a critical advantage if discrepancies appeared.
Phase 3: Cleanup
After sustained monitoring with no mismatches, the old-system shadow job was removed, leaving the new system as sole producer.
Tooling for Data Quality
Meta built a custom data quality analysis tool to catch edge cases across thousands of jobs. For each landed shadow partition, the tool read the corresponding production partition and compared row counts and checksums. Any mismatch was logged to Scuba, Meta’s real-time analytics data store. Hourly, the tool queried Scuba to identify example rows behind each mismatch and logged debugging details back—giving engineers a fast path to root-cause analysis. This same tool remains in use for release validation.
Rollout Signals and Rapid Rollback
Both systems rely on change data capture (CDC). Each job maintains internal full-dump and delta tables feeding a target table, with all metadata managed centrally.
CDC has a compounding risk: problematic landed data corrupts subsequent data. The mitigation focused on two fronts: catching issues before they reached data consumers, and stopping bad-data propagation quickly.
Early Signals After Rollout
Once a job entered reverse shadow, engineers triggered backfills on both the old and new production jobs. Matching backfill results signaled a successful migration. Any divergence triggered immediate rollback, before data consumers saw impact.
Halting Bad Data Spread
During reverse shadow, any partition flagged with data quality issues was marked in its metadata. For a delta partition, new data landed stopped and an alert fired. For a target partition, the system merged an older clean partition with new deltas instead.
This approach quickly isolated bad data. Rollback was simplified: query metadata for all flagged partitions and repair them via backfill.
Automation Across Tens of Thousands of Jobs
With migration volume that large, manual oversight was impossible. Meta automated the full lifecycle using job status signals continually sent to Scuba. External migration tools monitored these signals and automatically promoted or demoted jobs between lifecycle stages based on the defined criteria. System-level and job-level dashboards let engineers track aggregate progress and debug individual jobs.
Sequencing Migrations Under Limited Capacity
Shadow capacity was finite—not all jobs could be tested simultaneously. Job selection for each batch became a key efficiency lever. Jobs were categorized by throughput, priority, and known special cases. Preparation included:
- Excluding jobs with unresolved known issues to reduce noisy duplicates.
- Prioritizing based on business need and notifying dependent teams ahead of time.
- Holding back any job potentially affected by an active issue until the fix landed.
CDC's first snapshot—a full dump—is expensive to land and re-land when corrections are needed. Delaying shadow creation for jobs with known issues avoided redundant full dump work. A further optimization reused snapshot partitions already produced by the old system as an initial snapshot, reducing full dump load entirely.
