Why web experiments often fail
Running experiments on the web has historically meant compromising either user experience or data quality. Client-side rendering forces users to wait through loaders and layout shifts before seeing the experimental variation. Server-side rendering slows response times while experiments are evaluated on each request. Both approaches introduce enough latency that users behave differently than they would on a normal page load, skewing the very data you're trying to collect.
This tradeoff is avoidable. By pre-rendering each experiment variation as a static page and serving the correct one with a fast rewrite, you can run experiments that load instantly, shift nothing, and give you clean behavioral data.
Design goals for a zero-CLS experiment engine
We built our engine around a few non-negotiable requirements: no impact on end-user performance, automatic assignment to consistent experiment buckets, analytics events sent only with user consent, and a simple developer interface that doesn't require deep expertise in experimentation.
Fast reads with Edge Config
Experiment evaluation rules need to be available at request time, as close to the user as possible. Vercel Edge Config is a JSON data store built for this: reads complete within 15ms at P99 and can be as fast as 0ms, and its read speed doesn't degrade with geographic distance.
We use Statsig's Edge Config integration to keep experiment rules automatically populated in our project's Edge Config. Fetching our experiments in code requires only that stored configuration:
import { createClient } from '@vercel/edge-config'
import { EdgeConfigDataAdapter } from 'statsig-node-vercel'
import Statsig from 'statsig-node'
async function initializeStatsig() {
const edgeConfigClient = createClient(process.env.EDGE_CONFIG)
const dataAdapter = new EdgeConfigDataAdapter({
edgeConfigClient: edgeConfigClient,
edgeConfigItemKey: process.env.STATSIG_EDGE_CONFIG_ITEM_KEY,
})
await Statsig.initialize(process.env.STATSIG_SERVER_KEY, { dataAdapter })
}
async function getExperiment(userId, experimentName) {
await initializeStatsig()
return Statsig.getExperiment({ userId }, experimentName)
}
Building the engine with Next.js
Our Next.js-based approach combines several capabilities: type-safe experiment definitions, dynamic routes for pre-rendering variations, and Middleware rewrites for serving them.
Type-safe experiment definitions
To make experimentation predictable for developers across the monorepo, every experiment is defined in one typed, centralized source of truth. Each definition lists every possible value for its parameters, along with which paths the experiment should run on. The first value in each array is the default.
export const EXPERIMENTS = {
pricing_redesign: {
params: {
enabled: [false, true],
bgGradientFactor: [1, 42]
},
paths: ['/pricing']
},
skip_button: {
params: {
skip: [false, true]
},
// A client-side experiment won't need path values
paths: []
}
} as const
Pre-rendering page variations
We encode experiment values into the route path using dynamic routes, then pre-render each variation at build time via wrapped versions of getStaticPaths and getStaticProps. Each generated path holds a hash of the experiment values in its parameter slug. For pages running multiple experiments, the Cartesian product of all variations could balloon build times, so we use a generator function to calculate only the first n combinations (100 by default). Pages beyond that fall back to Incremental Static Generation or Incremental Static Regeneration, depending on whether you provide a revalidation interval. The tradeoff is between build duration and the delay an initial visitor might see on an incrementally rendered page.
The getStaticProps wrapper decodes the parameter from the URL to recover the experiment values for the page. Its signature matches that of a standard getStaticProps, so the interface stays familiar:
// Your encoding implementation
import { encodeVariations, decodeVariations } from './encoders'
export function experimentGetStaticPaths(
path,
maxGeneratedPaths = 100
) {
return (context) => {
const paths = encodeVariations(path, maxGeneratedPaths)
return {
paths,
fallback: 'blocking',
}
}
}
export function experimentGetStaticProps(pageGetStaticProps) {
return async (context) => {
const { props: pageProps, revalidate } = await pageGetStaticProps(context)
const encodedRoute = context.params?.experiments
// Read from URL or use default values
const experiments = decodeVariations(encodedRoute) ?? EXPERIMENT_DEFAULTS
return {
props: {
...pageProps,
experiments
},
revalidate
}
}
Implementing an experiment for a page then requires minimal code:
export const getStaticPaths = experimentGetStaticPaths("/pricing")
export const getStaticProps = experimentGetStaticProps(async () => {
const { prices } = await fetchPricingMetadata()
return {
props: {
prices
}
}
})
Serving variations with Middleware rewrites
Returning users are served their assigned variation via a cookie. New users without a cookie are assigned valuable parameters by reading experiment rules from Edge Config inside Middleware, then encoding the assigned values into the rewrite target route. A user requesting /pricing receives /pricing/0p0v0 internally while the browser bar remains unchanged. The cookie value comes directly from Statsig's Edge Config integration, which assigns users into consistent experimental buckets automatically.
import { NextResponse, NextRequest } from 'next/server'
import { get } from '@vercel/edge-config'
export async function middleware(request: NextRequest) {
if (await get("showNewDashboard")) {
return NextResponse.rewrite(new URL("/new-dashboard", request.url))
}
}
Since the page was fully rendered on the server, any React components using the engine's useExperiment hook already receive their assigned values without client-side evaluation — so no layout shift can occur.
Capturing analytics
With the correct variation rendering, a React Context wrapping the application reads the current experiment values and automatically records an EXPERIMENT_VIEWED event to our data warehouse, but only when analytics consent is present.
export function trackExperiment(experimentName) {
analytics(EXPERIMENT_VIEWED, getTrackingMetadataForExperiment(experimentName))
}
const Context = createContext()
export function ExperimentContext({
experiments,
path,
children
}) {
useEffect(() => {
for (const experimentName of getExperimentsForPath(path)) {
trackExperiment(experimentName);
}
}, [])
return (
<Context.Provider experiments={experiments}>
{children}
</Context.Provider>
)
}
Handling client-side experiments
Experiments that surface after the initial page load — like those rendered in a modal — need a different path. For these we read the session cookie directly from a client-side React hook to determine the assigned bucket and render accordingly. Since the page is already loaded, there's no need for routing, static generation, or server-side rendering for these cases.
Keeping assignments current
A cookie-based assignment can drift out of sync as experiment configurations change. For projects running today, the recommended pattern is request-time bucketing in Routing Middleware with background refresh handled by Vercel Functions and Fluid compute, while Edge Config remains the source for experiment configuration reads. This keeps the cookie-freshening logic working without opening clients to stale experiment assignments.
The result
The engine combines static generation, typed definitions, and fast edge data lookups into a developer workflow that ships experiments without the latency penalties of previous approaches. It gives you:
- Zero layout shift caused by experiment loading
- Support for many simultaneous experiments across pages and components
- Safe iteration and faster shipping for feature teams
- Behavioral data that reflects real user responses, not distorted by slow load times
