Why Figma left ECS behind
By early 2023, Figma had all of its services running in containers on AWS Elastic Container Service (ECS). That setup worked, but as the company’s infrastructure team looked at the next generation of what they call the compute platform—the layer that lets product teams own and operate services—they kept hitting ECS’s limits. The question became whether they were iterating toward a local maximum rather than the global one.
Importantly, Figma does not see itself becoming a microservices company. A small set of powerful core services provides modularization and traffic isolation, and new products are often supported by adding logic within those existing services. That made a Kubernetes migration digestible: there weren’t thousands of services to move.
Concrete gaps in ECS
Several missing features drove the decision. Service teams with complex needs were spending significant engineering hours working around ECS limitations. The most painful example was running etcd, the strongly consistent consensus data store, on ECS. Since ECS lacks Kubernetes’ StatefulSets (which give pods persistent identity), the team wrapped custom code into the etcd container startup to dynamically update cluster membership. The code was fragile and hard to maintain; on Kubernetes, StatefulSets handle stateful network assignment natively.
ECS also had no straightforward way to run services defined as Helm charts, the common packaging format for open source software on Kubernetes. Teams increasingly wanted to run OSS like Temporal, a workflow execution tool, but installing and maintaining those on ECS meant manually porting every service into Terraform definitions.
Smaller friction points accumulated too. Gracefully terminating a single poorly behaving EC2 instance is awkward on ECS, but on Amazon’s Elastic Kubernetes Service (EKS) you can cordon the bad node and let the API server move pods off while respecting shutdown routines.
The CNCF ecosystem draw
Beyond missing functionality, staying on ECS meant missing the open source technology in the Cloud Native Computing Foundation (CNCF) ecosystem. Auto-scaling was a top priority: Figma did not auto-scale any containerized services and was paying to keep everything provisioned for peak load even at night and on weekends. Kubernetes offers robust open source options like Keda, which scales not just on CPU utilization but on SQS queue length and custom Datadog metrics.
The team also anticipated adopting a service mesh. Traffic between services ran through AWS Application Load Balancers and Network Load Balancers, which had drawbacks: registering and removing targets takes minutes on NLBs, slowing emergency deploys and increasing mean time to remediation. Envoy is more customizable and supports custom filters—Figma had already built a standalone Envoy proxy cluster in front of a major service to shed load during incidents. Running Envoy as a mesh across the fleet pointed toward EKS and open source options like Istio rather than rebuilding that functionality on ECS.
The broader CNCF ecosystem is also evolving faster than AWS is likely to invest in ECS, given Kubernetes’ large, vendor-agnostic user base.
Benefits of being mainstream
Figma deliberately avoids being the biggest user of any service; those users hit rough edges and scaling limits first. On Kubernetes, many large companies run huge compute platforms, de-risking the platform for everyone else. Running on Kubernetes also reduces vendor lock-in. EKS provides a middle ground: a vendor-supported control plane, but with all services written generically for Kubernetes, moving to another vendor or self-hosting would not be a heavy lift.
Finally, hiring is easier. Engineers with prior Kubernetes experience bring context to decisions that would otherwise be novel, letting them hit the ground running.
Choosing what moves (and what doesn't)
Our core hypothesis was that migrating from ECS to EKS would leave Figma's compute platform in a materially better place, provided we could scope the work tightly enough to finish in a reasonable timeframe. The guiding principle was to keep the abstraction identical for every user of the platform: everything would run on EKS instead of ECS, but nothing about how services run, deploy, or are interacted with would change. This discipline matters because seemingly nonfunctional changes often produce second-order effects, and those indirect consequences are what typically derail migration timelines.
Two exceptions justify deviating from a strict parity scope:
- When recreating existing behavior on the new system would take more effort than absorbing the second-order effects. EKS is largely a superset of ECS functionality, so we rarely hit this case.
- When a decision is a one-way door or expensive to reverse later. In those situations, it is worth adopting the new approach from day one.
Improvements deliberately included
Even with a tight scope, we identified a few wins worth building in from the start.
Developer experience. On ECS, developers defined services through Terraform. Applying it created an empty ECS task set as a template. The developer then had to deploy the service, clone the template, substitute in the image hash, and deploy again with live instances. Adding an environment variable meant writing Terraform, applying it, then running a deploy—in that exact order, a sequencing requirement that frequently led to bugs when forgotten.
On EKS, we consolidated service definition into a single Bazel configuration file. Our CI tooling generates the Kubernetes YAMLs (including Ingress objects) on commit, and our in-house deploy system applies them. This change qualified under both exceptions: preserving the old Terraform workflow would have been substantial work, and keeping it would have complicated future migrations away from it.
Reliability. We run three separate EKS clusters, all actively receiving traffic for every service. Cluster-by-cluster operations reduce a full outage to at most a third of service impact, which is often minimal for users when requests can be retried or handled asynchronously. We have already seen multiple incidents where this setup meaningfully reduced blast radius.
This choice added real complexity to deploy pipelines and operations, but migrating directly to a three-cluster architecture was easier than bolting it on later, and it aligned with the infrastructure organization's primary goal of improving Figma's reliability.
Cost efficiency. We kept complex cost work out of scope, with one exception: node autoscaling. Our ECS services on EC2 were over-provisioned to handle surge capacity during deploys. Supporting dynamic node scaling from the start let us eliminate that waste for relatively little effort, using the open source CNCF project Karpenter to scale nodes up and down on demand.
Work deferred to fast-follows
Two obvious improvements stayed out of the initial scope. Our logging pipeline was expensive, routing logs from Cloudwatch through a transforming Lambda to Datadog and Snowflake. We considered replacing the forwarder with Vector as a sidecar in the EKS stack but decided the risk of porting all the transformation logic into Vector configuration was not worth upscoping the migration. Pod-level autoscaling was likewise deferred because it would add too much complexity.
Both became fast-follow projects we tackled recently, running in parallel with ongoing service migrations. This pipelining delivered incremental value without expanding the migration's risk profile.
Executing a safe migration
With a deliverable feature set defined, we focused on how to move a stable ECS stack onto a newer, less-tested platform without disrupting users. The techniques themselves are conventional, but they deserve emphasis.
Load test early
We built a "Hello, World" service and scaled it to the pod count of our largest production services. This exposed scaling problems in the platform's core compute services before real traffic arrived. Kyverno, the cluster security policy tool, is a good example: undersized, it noticeably slows pod startup.
Establish incremental rollback
Weighted DNS entries let us shift traffic gradually from an ECS service to its EKS equivalent, and shift it back just as finely. Unexpected impacts surface at unknown inflection points, so having granular control over the traffic surface and the ability to revert quickly was essential.
Put real workloads on it early
Staging cannot teach you everything. We migrated one service before finishing the staging environment—a decision that paid off by validating end-to-end workload execution and surfacing bottlenecks and bugs far earlier than otherwise possible.
Shield users from raw YAML
Direct Kubernetes YAML authoring is confusing. We defined a golden path for service owners, allowing customization only where special cases genuinely require it. Enforcing consistency by default saves users time and simplifies maintenance and future platform changes.
Collaborate with service owners
Service owners know their health signals best, so we partnered with them on monitoring and alerting updates while handling the new service configuration ourselves. Extensive conversations about tradeoffs before starting won buy-in early and ensured service owners saw the value in participating in a complex process.
Staff for the unexpected
A migration of this scale brings novel platform bugs, complex interactions, and deep debugging challenges. We staffed a team with the technical depth to handle those problems as they arose.
Timeline and outcome
The plan came together in Q1 2023. Q2 brought a staging environment and the first migrated service. Q3 focused on productionalizing, load testing, and preparing more services. Through Q4 and into early January, we cut traffic over progressively.
By January 2024, most of our highest-priority services ran on the new EKS clusters, including our core monolith, the multiplayer service that handles collaborative file edits, and the suite behind our Livegraph real-time update pipeline. The benefits are already visible: we no longer over-provision for deploys, we run across three clusters for reliability, and developer ergonomics have improved.
The migration proceeded with only minor incidents and little customer impact, and the three-cluster architecture deserves much of the credit. In one incident, an operator error destroyed and recreated CoreDNS on a production cluster. Previously, that would have meant a full outage. Instead, impact was limited to a third of requests, and most downstream services succeeded on retry.
Post-launch cleanup
Our debugging and operations tooling let service owners inspect running instances, shell into containers, and perform emergency actions. Shortly after launch, users told us that tooling was not friendly enough.
Two changes had made it complex. First, three clusters meant users had to run commands across each cluster and append cluster names to every invocation. Second, the migration introduced granular RBAC-based permissions in partnership with our security team, honoring least privilege—but requiring users to know which roles they held and which they needed. Both issues were straightforward to fix. We paused other work and updated the tooling to infer the correct cluster and role automatically, sparing users from chasing down permissions during a 3 a.m. incident.
Roadmap: beyond the migration
With the bulk of the migration complete, we’re shifting our attention to parallel workstreams: finishing the move for remaining services while making the platform itself more efficient and cost-effective. Our immediate focus includes simplifying the logging pipeline design, enabling horizontal pod autoscaling through Keda, and porting our most expensive service to Graviton processors. The Graviton work delivers near-term cost savings and establishes a proven path for other services to follow.
We see significant headroom in areas we haven’t yet been able to invest in deeply. Our networking stack is a prime candidate for reliability and observability improvements, likely through a service mesh. We’re also investigating whether we can consolidate our infrastructure code by moving more resources out of Terraform and into AWS Controllers for Kubernetes (ACKs), which would further unify our stack. Finally, we’re partnering with our developer experience team to align how services are run in local development with how they run in production environments.
These efforts align with our broader goal of building long-term, sustainable infrastructure solutions across the company.
