Where the legacy sync broke down
Figma's first synchronization pipeline, built in 2020, was simple by design: a daily cron job ran SELECT * FROM <TABLE>, pushed the result to S3, and loaded it into Snowflake. That held up while tables were small, but growth in users and data exposed the limits. By 2023 a daily sync took roughly six hours, extra database replicas had to be maintained to serve the exports, and the largest tables took several days or longer.
The team weighed three paths. Staying on the legacy process was untenable: delays aside, the extra replicas cost millions of dollars a year. Adding parallelism was considered as a quick fix but did not scale. The remaining option — a full overhaul around incremental synchronization, which captures and applies only recent changes rather than re-transferring whole datasets — meant a larger, longer investment but the most scalable result.
Why the pipeline was built in house
Incremental synchronization depends on three capabilities: table snapshots, change data capture (CDC) streams, and incremental merge. After evaluating proprietary end-to-end products, the team found none that fit on flexibility, cost, and scale.
- Flexibility: generic SQL-compatible tools ignored vendor-specific capabilities. APIs for Amazon Relational Database Service (RDS) for PostgreSQL, for instance, could produce snapshots without a separate replica, but the off-the-shelf options did not use them.
- Cost: proprietary solutions were projected at five to ten times the price of an in-house build at Figma's scale.
- Scale: many tools could not keep pace with current and anticipated growth, and building internally preserves the ability to adapt quickly.
The bespoke approach let the team combine lower-level open-source and managed components: Amazon RDS exports snapshots to S3; Kafka Connect, hosted on Amazon Managed Streaming for Apache Kafka (MSK), handles CDC once paired with a Snowflake Connector; merge logic is implemented as Snowflake stored procedures driven by Snowflake tasks.
Design principles
Four principles shaped the new architecture: reduce end-to-end synchronization latency, keep costs low as data grows, stay compliant with data regulatory standards, and guarantee data integrity — accurate, complete, consistent, and trustworthy throughout the lifecycle. They translated into two workflows: a bootstrap workflow that brings new tables into the pipeline and a validation workflow that verifies correctness as data flows through.
Onboarding a table
- The existing in-house CDC service — already wired into the database topology — captures the new table from Postgres and publishes events to per-table Kafka topics.
- Amazon RDS exports the latest daily snapshot to S3; duration depends on table size.
- Snowflake's
COPY INTO <table>loads the S3 data into dedicated per-entity base tables. - A Snowflake Sink Connector in MSK Connect streams Kafka topics into per-entity CDC tables, with the Kafka start offset set before the snapshot timestamp.
- A scheduled Snowflake task periodically runs a custom
MERGEstored procedure. - Once synchronization catches up to recent changes, a lightweight view over the base table is created for user queries, completing onboarding.
Re-bootstrapping, needed for events such as schema evolution, runs with zero downtime: all bootstrap artifacts except the final user-facing view are versioned, so a new bootstrap proceeds in parallel with live operations and is promoted through an atomic view update.
Cell-level validation
Partial failures, misconfigured components, software bugs, or anomalous source data can corrupt a pipeline at any stage — snapshot export, CDC capture, or incremental merge — and surface as silent inconsistencies. The validation workflow guards against this: it clones the live base table as the source, runs the bootstrap workflow with automated merges disabled to write into a temporary schema as the target, and aligns both to the same point in time using the exported CDC data. It then performs cell-to-cell comparisons, feeding the results into monitoring and alerting. Because the check is cell-level and CDC-aware, the team gains confidence in data integrity before and after launch.
Automation in two layers
Orchestration across many network calls and dependencies is handled with AWS Step Functions, split into two categories. First-level workflows run ad hoc from just an entity name, handling bootstrap or validation without manual intervention unless monitoring raises an alert — alarms loud enough to force immediate action, with clear remediation steps to prevent recurrence. Second-level workflows watch state and trigger the first level on a schedule:
- a controller checks every few hours for entities to onboard or re-bootstrap;
- a validation dispatcher launches validations weekly for each table;
- an invalidator re-bootstraps each table weekly to protect integrity;
- a sanitizer cleans up orphaned artifacts weekly.
Testing and launch
Testing was deliberately aggressive: in staging, an automated routine re-bootstraps every table weekly to surface problems early. It paid off within a week, when testing caught a severe failure mode that would have caused a site-wide outage of at least twenty minutes in production. That result reinforced full automation as the goal — where complete automation seemed risky, the team automated partially and worked incrementally toward the rest. Since launch, operations have run without major incidents.
Operating the new pipeline
The architecture's added flexibility and automation allowed Figma to build features that target freshness, manual control and change visibility.
Tuning freshness and manual syncs
End-user feedback shaped the default merge frequency, which was set to every three hours to balance baseline freshness across all entities against Snowflake compute costs. Tables needing tighter latency can override that default; the billing pipeline uses half-hour overrides and saw a considerable reduction in end-to-end pipeline latency.
Because merge jobs are queued, merges can be triggered safely at any time. That guarantee underpins a CLI tool for immediate, manual synchronization outside the automated schedule, giving teams on-demand access to fresh online database data in Snowflake.
Change data capture inside the warehouse
CDC data was already being imported into Snowflake for internal purposes, so Figma exposed it to end users who want the sequence of changes behind an entity's state rather than only its current values. During incident response it offers a secure offline environment for debugging unexpected database writes—for example, querying all email insert, update and delete events for users on a given team over the past week. Combined with sync-on-demand, that data can be queried in Snowflake in near real time. To comply with retention policies and prevent unbounded storage growth, CDC records are purged automatically after a predefined period.
Outcomes
- Freshness: data previously ran 30 or more hours old; it is now three hours old or less, and can be configured down to minutes.
- Scale: the pipeline reliably handles tables more than ten times larger than before, with consistent, predictable performance as Figma grows.
- Developer productivity: before launch, the team interviewed developers to identify their needs and integrated the pipeline with systems the team already knew, building confidence against workflow disruption. The result was a significant productivity boost from lower operational overhead and near real-time access to online data in the analytics warehouse. Current state and change history can both be queried, fresh within minutes.
- Cost: support for horizontally sharded databases was prioritized early, since those databases have fewer tables but use more database machines, each with its own batch replica. The pipeline now delivers multimillion-dollar annual savings by optimizing infrastructure and resource utilization, eliminating redundant processing and scaling with the business.
What comes next
- Fully automated onboarding: adding tables currently requires a pull request against an allowlist. Wiring database topology directly into the pipeline would remove that friction and the manual overhead.
- Point-in-time table support: CDC data could answer queries about a table's state at any position within the retention window, improving debugging, incident response and analytical flexibility.
- Incrementally refreshed downstream models: many downstream analytical models still run as traditional batch processes; the new pipeline would let them refresh incrementally, improving efficiency and cutting latency across the analytical workflow.



