A search for answers

Earlier this year, Figma's search team began investigating why search had become slower and less reliable as the platform scaled. The goal was to build a foundation that could support future growth, but the investigation quickly revealed that the problem wasn't where anyone expected it to be.

Until late 2023, Figma relied on an older version of ElasticSearch. The team then began migrating to OpenSearch, a fork of ElasticSearch created after ElasticSearch's license change in 2021, running on AWS's managed OpenSearch service. While the two are mostly compatible, small differences accumulated over three years made the migration harder than anticipated.

The latency contradiction

Figma's monitoring, via DataDog's native OpenSearch integration, reported that the average search took about eight milliseconds. That seemed improbably fast for searching through terabytes of data spread across hundreds of index shards. Yet Figma's search API had a 99th percentile latency of nearly one second. Something didn't add up.

The team added metrics and traces around the major blocks of internal search code to find where time was actually being spent. The data revealed several key insights:

  • OpenSearch reported an average latency of 8 ms, but Figma's calls to their API library saw an average latency of 150 ms, with a 99th percentile of 200–400 ms. The minimum latency was over 40 ms—higher than OpenSearch's reported maximum.
  • Considerable time was spent building queries before they were even sent to OpenSearch.
  • Even more time was spent on permissions checking after results returned, to ensure users never saw files they couldn't access.
  • Performance was unstable, varying greatly from hour to hour and day to day, with peak times hundreds of milliseconds slower than weekends.

The contradiction seemed impossible: OpenSearch and Figma's code were running in the same AWS availability zone, only a couple of milliseconds apart. Digging into the documentation revealed the answer: the 8 ms metric was the single-shard average query time, not overall query time.

How OpenSearch reports time

When OpenSearch receives a query, it hits a coordinator node, which sends copies of the query to worker nodes—one per shard for the index being queried. Nodes typically take turns handling coordinator duty. This is the "query" phase. The coordinator then collects results, sorts them, and requests more information from the shards that returned the best results—the "fetch" phase—before returning results to the client.

The 8 ms metric only covered each individual per-shard query between coordinator and worker nodes. With Figma's initial configuration, a single user query could spawn up to 500 per-shard queries. Many ran in parallel, but not all. That was the source of the discrepancy.

After consulting AWS, the team learned that none of OpenSearch's metrics or logs track overall query time—only per-shard time. The sole place OpenSearch reports overall query performance is in the query API response, in a field called took, which gives the number of milliseconds OpenSearch took to answer the query. Figma parsed this value from each search response and added it to monitoring, giving a backend latency number that mostly matched their own timing wrappers.

The real bottleneck

The reconciled data told a surprising story: less than 30% of total query API time was spent waiting on OpenSearch. Pre-processing and post-processing took more time than the actual search.

Pre-processing fetches information about the files a user can access and builds an OpenSearch filter clause that mostly excludes files they can't. Post-processing verifies that the user actually has permission for every file returned. Both steps were slow, and post-processing particularly so.

Working with the permissions team, the search team analyzed that code. Statistical analysis showed that evaluating parts of the permission system in a different order could produce identical results much faster. They also discovered a shocking amount of time was being spent on runtime type-safety checking in Ruby within the permissions system; disabling the most intrusive parts of it yielded substantial speedups.

Slow search traces also revealed oddly slow database queries. The database itself was fast, and the load-balancing proxy was fast, but issuing queries sometimes took tens of milliseconds. After examining source code and traces, a team member identified a problem with how new database connections were set up in new threads. The connection pool wasn't large enough, causing expensive setup and teardown operations inside each thread that were never needed. Fixing that produced substantial speedups not just in search, but across all of Figma.

With that insight, the team re-evaluated past threading experiments where parallel database reads had rarely been a performance win. With the new initialization code, nearly every opportunity to issue queries in parallel made Figma faster.

Digging into measurements

With better metrics and fixes for the biggest issues in place, the Figma search team turned to deeper questions guided by data: whether OpenSearch queries were efficient, whether the indexed data was right, and whether the OpenSearch cluster was configured correctly.

Query efficiency

Using OpenSearch’s query profiler, the team found that most queries touched only a few hundred documents per shard, despite indexes containing millions of documents per shard. Filtering in the pre-processing step eliminated most files users couldn't access, and OpenSearch’s query optimizer leveraged that effectively. Queries were not the bottleneck.

Index data quality

Whether the index contains enough useful data is partly a search-relevancy question, not strictly a performance one. The team continuously runs experiments on relevancy, but they did determine that most indexed data wasn't useful. Trimming index size by 50%, and then by an additional 90%, had no measurable impact on relevancy. Smaller indexes made everything faster, easier, and cheaper.

Sizing and configuration: where defaults fail

OpenSearch’s flexibility creates complexity. AWS lists 139 OpenSearch server instance types, ranging from $0.02 to $17 per hour. Shard count, compression types, and search concurrency all allow CPU, memory, and disk tradeoffs — none with an obvious "best" value. AWS provides sizing guidance focused on shard size and the ratio of shards to nodes, but that guidance is built for throughput-heavy log querying, not latency-sensitive document search.

To understand how these options affected search performance, the team built a load testing system with non-production OpenSearch clusters, loading data and running queries to measure impact. OpenSearch’s own benchmarking tool, opensearch-benchmark, didn't fit the use case: it's meant for regression testing during OpenSearch development, struggles with massive numbers of randomized queries against existing instances, and uses client-side latency rather than the server-side "took" metric. A custom benchmarking tool written in Go in an afternoon gave repeatable results.

What the tests revealed

  • Too many shards. Reducing from 450 to 180 shards — a 60% drop — increased the maximum query rate before excess latency by over 50%. P50 latency also decreased with fewer shards, since the coordinator had less collection work per query.
  • Less index data meant more consistent performance. The initial 50% size cut lowered query latency and made it more consistent; the further 90% reduction made the entire dataset fit in the operating system’s disk cache, improving cache hit rates and predictability.
  • AWS’s sizing recommendations didn’t apply. They suggest keeping shards under 50 GB and provisioning one shard per 1.5 CPUs. That suits log-like workloads, but for document search it pushes coordinators to manage too many shards per query. Effective filters meant better performance came from fewer, larger shards.
  • Imbalanced node specs. The cluster was provisioned with too much CPU and too little RAM. Switching to nodes with one-third the CPU and 25% more RAM cost about half per node and gave slightly better performance — even before index sizes were reduced.
  • Zstandard compression was neutral. Not a major win, but no downside either.
  • Concurrent segment search never helped. It added a few milliseconds of latency at low query rates, and latency rose faster with load — surprising given abundant free CPU and the expectation that more parallelism would help.

Overall, the team cut API latency by about 60%, raised maximum queries per second by at least 50%, and reduced total cost by over 50%. The work spanned monitoring, bug fixes, index size reduction, and configuration tuning. No single change was decisive, but the combined effort improved search performance and positioned Figma’s search infrastructure for future growth.