How Figma's AI Search Is Built

At Config 2024, Figma introduced AI-powered search across designs and published components. The new features let users find assets through screenshots, layer selections, or natural language descriptions, rather than relying on exact names or manually curated keywords.

Two search flows are involved. Search for designs indexes frames across files, making it possible to locate unlabeled frames in large, multi-page documents. Search for components extends the Assets panel with semantic understanding; a component called 😀 can be found by searching "smiley," "happy," or "grin," and both flows support visual queries via screenshot or selection.

The foundation of both features is a multimodal embedding model that maps text and images into the same numerical space. Figma uses the open source CLIP model. An embedding for the string "cat" will be numerically close to an embedding generated from an image of a cat, enabling image-to-text and text-to-image comparisons. All models were fine-tuned exclusively on public, free Community files containing UI images; no private Figma files or customer data were used in training.

Search works by generating embeddings for indexable content—components, frames—storing those vectors, and then comparing a query embedding against them via nearest neighbor search. Generating a query embedding depends on the input type: text or screenshots are passed through the model directly, while a selection of Figma layers is first rendered as a screenshot before embedding generation. Early experiments with converting layer selections to text (such as JSON) produced worse results, so Figma standardized on the image-based path for both query types.

Building the index

Populating the search index requires discovering indexable content inside Figma files. Unpublished frames are not readily enumerable, so Figma runs a headless, server-side version of its C++ editor in an asynchronous job to identify all frames in a file. That frame identification job stores metadata in DynamoDB, which suits the features' simple key-value access patterns with high throughput and no transactional requirements. Thumbnails are rendered and uploaded to S3. Each stage of the pipeline is separated into discrete jobs, which gives finer control over batching and retries.

Embedding generation happens in AWS SageMaker. Thumbnail URLs are sent to a SageMaker endpoint in batches for parallel inference, and the job parallelizes image downloading, resizing, and normalization. Finding the right batch size required experimentation: past a certain threshold, latency grew linearly rather than sublinearly with batch size. Generated embeddings are persisted before the final pipeline step runs.

The last step writes embeddings to OpenSearch—already widely used across Figma for traditional search. Each entry includes metadata such as frame name, containing file, project, team, and organization, which supports faceted filtering alongside the vector search itself.

Component search and result blending

Publishing a component library triggers an asynchronous job that computes embeddings for each component thumbnail. The model here is similar to the design search model, but fine-tuned on publicly available Community UI kits. Like the design index, this model saw no private file or customer data.

Component search historically relied on lexical fuzzy string matching over names and descriptions. Rather than replacing that index, Figma queries both the lexical and embedding indexes in parallel for every search. Since scores from the two OpenSearch indexes are not directly comparable, Figma normalizes both result sets using min-max scaling and boosts exact lexical matches. Interleaving results by these updated scores preserves lexical precision while adding semantic recall, so searching "mouse" surfaces not only assets explicitly named "Mouse" but also cursor-related icons.

Controlling the cost of indexing at billion-entry scale

Producing embeddings and indexing entries across Figma is an expensive, time-consuming task — and it's made harder by how the product is organized. Because search only works correctly for a user once their entire team's data is indexed, onboarding even a small test group requires processing whole teams. Since Figma has a long tail of small teams, enabling AI search for even a modest slice of users effectively means indexing nearly all of the company's data. That puts a premium on keeping both the initial backfill and ongoing indexing cheap.

Interestingly, the dominant compute cost in the pipeline wasn't embedding generation at all. The expensive parts were identifying meaningful designs inside Figma files and generating thumbnails for them. The team focused optimization efforts there:

  • Replacing Ruby with C++. The original logic serialized each Figma file as JSON and parsed it in Ruby, which was slow and memory-hungry. A C++ rewrite that skips intermediate serialization cut runtime and memory dramatically.
  • Switching to software rendering. Thumbnails moved from GPU-based rendering on an older AWS instance type to CPU rendering with llvmpipe on newer hardware. The CPU machines are far cheaper and faster on this workload.
  • Debouncing freshness. The index was originally refreshed on every file change, but users tend to make many edits in one session. Data showed that capping refreshes to once every four hours meant processing only 12% of the data while keeping the index reasonably current.
  • Scaling clusters with traffic. Figma usage is strongly diurnal, so scaling down during low-traffic windows avoids paying for idle compute.

Shrinking the search cluster

OpenSearch — the second largest cost driver — needed a substantial cluster just to keep large indexes resident in memory. The team made two big changes to bring that down.

The first was redefining what counts as indexable. By dropping draft files, duplicate designs within a file, and copied files without changes, index size was cut in half. That also improved the product: search stops surfacing near-identical duplicates.

The second was vector quantization. By default, OpenSearch's kNN plugin stores each embedding element as a four-byte float. Quantization compresses those embeddings, reducing the memory footprint for storage and search in exchange for a small accuracy tradeoff.

OpenSearch quirks, and workarounds

Even on the latest OpenSearch release supported by AWS, the team ran into quirks worth documenting.

Non-deterministic replica queries. End-to-end testing showed search results varying periodically. After debugging, the cause turned out to be replicas returning different results than primaries, triggered by a casting error (Reader cannot be cast to class SegmentReader) deep in the delete path. Working with the AWS OpenSearch team, the issue was confirmed to affect clusters using segment replication; a fix was shipped in the k-NN plugin.

Removing embeddings from _source. To save storage and speed up queries, the team stripped the embedding vector from _source — the original document body OpenSearch returns on read but doesn't query. The problem: on document updates, OpenSearch pulls the existing document from _source, applies the diff, and rewrites it. With embeddings gone from _source, any update (say, a file rename) silently wiped the embedding. The fix kept the size optimization but re-fetched embeddings from DynamoDB whenever a document updates.

Status

AI-powered search in Figma is in early beta. The rollout will continue to expand over the next several months as Figma incorporates feedback from early users.