Meta's Facebook Feature Framework (F3) handles authoring, compilation and serving of machine learning features for ads ranking models, where a single request can demand more than 20,000 features. First presented at AI @ Scale 2020, the framework originally cut execution costs by compiling all features into one merged graph — but that same graph doubled as the release unit, so a single bad definition could stall the entire pipeline.

Why shared execution made releases fragile

Features in production machine learning are rarely computationally independent. A seven-day click count and a seven-day impression count may read the same event source and apply the same time-window filter; a click-through rate feature can then reuse both results. F3 modeled extraction as a directed acyclic graph (DAG): nodes perform individual units of work such as reading an input, filtering or aggregating, and edges express dependencies. Engineers write features independently in a high-level language, and the F3 compiler turns each definition into a small graph before merging every feature a service uses into one graph, collapsing duplicate operations into shared nodes. At runtime, extraction is a traversal of that merged graph.

The trade-off surfaced as scale grew. Historically F3 served a few hundred features and merges succeeded smoothly; with more than 20,000 today, one malformed definition could hold back all the others. Different feature types need their own validation, and new versus modified features are held to different bars — a feature already consumed by a model requires model-related tests that only run in a later, computationally expensive release stage. Under a single-release-unit model, every other feature waits for those tests. Incremental improvements to the merged graph restored release speed temporarily, but the gains eroded as more features landed.

F3 Lightning: independent releases, shared execution

F3 Lightning is an infrastructure layer built on F3 that decouples feature releases from shared execution. It makes two changes:

  • Deployment granularity — each feature is an independently releasable configuration, validated, shipped and rolled back without touching any other feature.
  • Efficiency parity — because per-feature releases remove the merged graph that provided F3's efficiency, execution was reworked through four optimizations: stable computation identifiers, a single shared execution plan, one shared memory buffer across all features, and dense index mapping for direct array access.

Isolating releases in the compiler

Each feature is compiled on its own into a single configuration carrying every node it needs, in order, with nothing borrowed from a neighbor. Configurations have no relationship to one another, so a problem in feature C does not stop feature A from shipping. Release becomes an ordered queue rather than an all-or-nothing batch, with changes prioritized by what they unblock: a new feature no model consumes yet can go out quickly on standard validation, while a change to a feature a live model reads gets extra testing first.

Early prototypes that separated only deployment used substantially more CPU than traditional F3, since each configuration arrived at the serving process as its own graph and re-executed every read, filter and aggregation the merged graph had eliminated. The remaining work was to regain shared execution without regaining the merge.

Restoring parity at startup

Rather than rebuilding a merged graph at startup, F3 Lightning consolidates the work that active configurations share and arranges it into a single execution order. Four steps closed the gap: one compiler prerequisite and three startup optimizations.

Content-derived identifiers

Deduplication requires the runtime to recognize identical computations across independently compiled configurations. Local node numbers cannot do this — the same seven-day filter might be node 2 in one configuration and node 7 in another, and inserting one node renumbers everything after it. Traditional F3 sidestepped the issue because a single compiler run saw every feature and could assign global identifiers.

F3 Lightning instead derives each identifier from what the work does rather than where it sits: the operation, its parameters, and the identifiers of its inputs. Because each input identifier already reflects everything upstream, the identifier captures the full chain of work producing the result. Inputs are placed in a standard order where order does not affect the result, and aggregations include defining properties such as the time window and grouping keys. Features defining the same operation over the same inputs therefore receive the same identifier even if compiled and released weeks apart, and any change in meaning changes the identifier.

A single plan, executed as a list

At service startup, F3 Lightning builds one execution plan across all active features, using a topological sort to place every node after its dependencies. Nodes with the same identifier are kept once and every consumer connects to that copy, so the computation runs once regardless of how many features use it. The startup pass then flattens the graph into an ordered list, letting each request run a single forward pass instead of walking pointers.

To contain failures, a computation that fails or lacks input causes everything downstream to be marked not ready and skipped; independent branches keep running, so a failure affects only its dependents.

One buffer per request

After deduplication, CPU use was still above traditional F3. The residual cost was memory layout: each configuration carried its own request-time data container, scattering intermediate results across allocations and costing a cache miss on every jump between them. F3 Lightning substitutes a single buffer per request, rebinding each feature's references to buffer slots at startup.

  • Locality — consecutive plan steps touch adjacent memory, restoring the sequential access pattern of the merged graph and eliminating per-feature allocation on every request.
  • One readiness vector — a single vector stored apart from the values lets the executor test readiness without reading the value. The marker must be separate because an empty list is a valid result, so "empty" cannot mean "not computed yet."
  • No duplicate copies — list and map values are not copied per consumer; the shared slot references the producer's result, so a computation used by 20 features is computed and stored once. Scalars are still copied, which is cheaper than following a pointer. Safety holds because the buffer lasts one request and execution order guarantees each result outlives its last reader.

Dense index mapping

Consolidated values were still expensive to find: every access resolved an identifier by hashing a key, probing buckets and following a pointer, which does not scale to Meta's serving volume. F3 Lightning assigns every unique identifier a consecutive integer slot during startup and rewrites all references to point at those slots, so identifiers are used only during initialization. Reads become direct array indexes into the shared buffer, with no hashing or lookup table on the request path.

Outcome

Comparable extraction features now reach production ~3.6x faster than on traditional F3, with no increase in infrastructure compute cost, and the shorter cycle allows more model iterations in the same period. Three lessons came out of the project:

  • Separate deployment and execution boundaries. When one artifact governs both how components run and when they ship, any single failure blocks everything. Splitting the roles dissolved the trade-off between efficiency and release speed.
  • Move invariant work to startup. Identifying shared work, determining execution order and locating results are identical for every request; doing them once at service start means independent releases add no per-request cost.
  • Identical operation sequences can still differ in CPU cost. Even after eliminating every repeated computation, CPU use stayed elevated because each feature kept its own copy of shared results. Consolidating to one copy per result closed the gap — where results live and how many copies exist can matter as much as how many computations run.

F3 Lightning extends Meta's F3 ads ML feature computation and serving platform, the same system whose development and productionization the following engineers contributed to.

Credits

The authors thank Sourabh Bansal, Ping Chen, Weitao Chen, Benjamin Dummitt, Junhua Gu, Henry Hu, Jing Huo, Joffrey Liu, Rocky Liu, Weixi Ma, Shuby Mao, Sandeep Pandey, Mihir Rastogi, Tim Sombuntham, Matt Steiner, Vivienne Sung, Ritwik Tewari, Vivek Trehan, Arnaud Venet, Ziheng Wang, Gautam Wilkins, Fan Yang, Qiao Yang and Jie Zheng, along with the entire team behind the development and productionization of F3, Meta's Ads ML feature computation and serving platform, and F3 Lightning, the capability built on top of it.