Why time-to-first-batch matters
For machine learning engineers, iteration speed is largely determined by time-to-first-batch (TTFB) — the interval between submitting a training job and the moment the first batch of data hits the model. TTFB sits on the critical path of nearly every ML workflow. It includes config validation, feature pre-processing, and infrastructure overhead like capacity queuing, and optimizations to any of those components can ripple through the entire training cycle.
At Meta's scale, TTFB frequently shifts subtly as developers modify models, launchers, or architectures. Two complementary capabilities are needed to keep the metric healthy: an offensive one — an experimentation framework that lets owners quantify the impact of proposed changes — and a defensive one — continuous regression prevention that catches TTFB degradations before they reach production.
AI Lab: an A/B testing framework for ML infrastructure
AI Lab is Meta's pre-production framework for continuously running common ML workflows as A/B tests. It measures how recent changes affect metrics like TTFB. Built on the same foundations as MobileLab, AI Lab serves dual duty: it can proactively validate improvements, and it automatically blocks regressions prior to release.
Constructing AI Lab came with constraints. GPU capacity is scarce, so the system had to be a net positive on capacity. AI Lab works with partners on shrunk models and simple configurations — some running on CPU only. An auto-shrinker ensures tests execute the same code and configurations as production while consuming fewer compute resources. It reduces training iterations and model size, and can enable more deterministic behavior. Most tests complete in under 10 minutes, which is well-suited for developers iterating on TTFB changes.
Case study: shipping Cinder with AI Lab
Meta's open source Python Cinder runtime delivered up to a 40% TTFB improvement through aggressive lazy imports. AI Lab was instrumental in validating and fine-tuning that rollout.
Offensive validation
Rather than experimenting on live ML engineer workflows — which can take days or weeks to validate a performance hypothesis — AI Lab lets developers test a proposed Cinder version against a comprehensive set of representative ML scenarios in under an hour. This enabled an iteration loop that yielded a 2x increase over the original TTFB improvements.
One example: profiling with Cinder showed up to 10% of execution time went to a workflow that merely pretty prints output. The memoization method in use triggered a repr() on an underlying data structure — which was huge in typical ML scenarios. Switching to an object wrapper and comparing memoized values via object identities solved it, and AI Lab confirmed the improvement before the change shipped.
Defensive regression catch
During the Cinder rollout, an unrelated regression appeared. An engineer added logging they believed to be asynchronous, but one nested client was synchronous, making the call blocking. AI Lab, using Incident Tracker, attributed the regression to the specific change. The change author was notified and reverted the commit before it reached production. The Cinder team never had to worry about conflating an unrelated regression with their rollout.
Scaling regression prevention
Benchmarking every ML scenario against every change at Meta is infeasible. Instead, AI Lab follows an approach similar to predictive test selection: cap the capacity used, and maximize the number of regressions and improvements caught as early as possible in the development cycle.
This plays out on two fronts:
- Per code change: Run relevant, effective, and computationally efficient (often CPU-only) AI Lab tests on prospective changes before code review even begins.
- Per release: Run a broader set of AI Lab tests before release, then perform a bisect-like attribution to identify the root cause when a regression appears. This fallback is effective for finding regressions that require more computationally intensive tests.
When a statistically significant change is detected via a t-test, AI Lab performs additional checks before flagging a regression or improvement:
- Confirmation runs to reproduce the result.
- A check that the effect size exceeds a dynamic threshold based on the test's standard deviation and a tuned receiver operating characteristic. A partner requiring fewer than one false positive per week, for example, sets the threshold that finds the most true positives while respecting that bound.
Open collaboration
While AI Lab is an internal-only tool, Meta is interested in hearing from others running similar platforms. Synthetic signal production benefits both developers — who can rapidly test hypotheses — and users, who experience fewer regressions. Meta is open to industry collaboration on improving tools like AI Lab and optimizing metrics beyond TTFB.
