How to Accurately Test Significance with Difference in Difference Models

Why Standard Tests Fail on Time Series

Difference-in-differences (DID) is a staple for estimating causal effects when A/B tests aren’t possible—for instance, when the intervention is an external shock or when true experimental controls are infeasible. The method is straightforward: compare the change over time in a treated group against the change in an untreated control group, subtracting out common trends so only the treatment effect remains. In practice, this is often fit as a linear model with an interaction term for treatment and post-period, which yields a coefficient estimate for the effect along with a standard error and p-value from standard software.

That standard output is the trap. DID relies on time series observations that are typically autocorrelated—each point is partly explained by its predecessors. This violates the independence assumption behind ordinary least squares (OLS) standard errors. Positive autocorrelation, the more common case in practice, inflates the apparent information content of the data. Standard routines become overconfident, reporting standard errors that are too small and p-values that are too liberal. The consequence: you declare significance far too often. In simulations with positively autocorrelated data and no true treatment effect, standard DID fitting at an alpha of 0.05 produced a false positive rate above 30%—six times the intended rate. Negative autocorrelation has the opposite effect, making tests overly conservative, though it appears less frequently in real-world data.

Remedies That Respect the Data Structure

Correcting for autocorrelation in DID significance testing requires methods that acknowledge the dependence between observations. Three practical approaches stand out, each with trade-offs in power and false positive control.

Averaging the Time Series

One instinctive fix is to collapse the time series for each unit into two averages—one for the pre-period, one for the post-period—and run the DID model on those aggregated points. This drastically reduces the data to something closer to independent observations, sidestepping autocorrelation entirely. It protects against false positives reasonably well, but the cost in statistical power is severe. By discarding within-unit temporal variation, you risk missing real effects, especially when the number of time points is small.

Clustered Standard Errors

A more sophisticated approach is to cluster the standard errors at the level of the time series unit (e.g., the show or the market). Clustering permits arbitrary autocorrelation within each unit and yields robust standard errors without throwing away data. This approach performs well in larger datasets, where the number of units is sufficient for the asymptotic theory behind clustered inference to hold. The benefit is a good balance between true positive rates and control of false positives—provided you have enough units to support the methodology.

Permutation Testing

A non-parametric alternative is the permutation test. Instead of relying on distributional assumptions, you repeatedly shuffle the treatment labels across units, recompute the DID estimate each time, and build a null distribution from the data itself. The observed effect is then compared against that distribution to obtain a p-value. Permutation testing is computationally heavier than the other two options, but it is highly robust and works well even when the number of units is small—precisely the situation where clustered errors and averaging struggle. In simulations, permutation testing delivered the best trade-off between detecting true effects and limiting false positives for datasets with few time series units.

Choosing an Approach

The practical takeaway from simulation experiments is that no single method dominates in all settings. The recommendation is to let your dataset size guide the choice:

• Small numbers of time series units: Prefer permutation testing. Its robustness to autocorrelation and limited unit counts makes it the most reliable choice for false positive control without excessive loss of power.
• Larger datasets: Clustered standard errors are superior, offering a stronger balance between statistical power and false positive rates while remaining computationally efficient.
• When protection against false positives is the highest priority: Averaging is broadly safe but tends to be underpowered—use it only when you are willing to trade sensitivity for safety.

The key is to never rely on the default standard error output from standard DID software for time series data. Regardless of which remediation you pick, you must explicitly account for the autocorrelation that is structurally present in your observations or risk making decisions on p-values that flatter your findings more than the evidence justifies.

Three Fixes for Autocorrelated Difference-in-Difference Data

With a false positive rate exceeding 30%, the naive approach to significance testing on autocorrelated data is untenable for product decisions. We evaluated three correction methods frequently cited in the literature—averaging, clustered standard errors, and permutation testing—to see which offers the most reliable inference across different data scales.

Averaging

The most direct way to eliminate autocorrelation is to remove the time series structure entirely. Averaging collapses each unit into a single pre-intervention and post-intervention mean, producing two data points per unit. This effectively turns the dataset into a cross-section, where conventional standard error formulas and significance tests become valid again.

In simulations, averaging brought the false positive rate back to the expected 5% threshold. The trade-off, however, is power: reducing thousands of observations to a handful of summary statistics dramatically shrinks the effective sample size, making it harder to detect genuine effects.

Clustered Standard Errors

Clustered standard errors keep the full dataset intact and require only that the model know which observations belong together. The cluster is simply the time series unit: for instance, your Spotify usage across days is correlated and forms one cluster, while another user's usage forms an independent cluster.

Standard ordinary least squares routines assume no correlation between any observations, which is why they understate uncertainty on autocorrelated data. The clustered approach instead estimates within-cluster correlation from the model residuals, then incorporates that correlation into the standard error. Positive autocorrelation inflates the residual error and, consequently, the standard error—precisely counteracting the overconfident estimates from standard output. For a deeper treatment of the mechanics, see Cameron and Miller (2015).

Simulation results confirm this behavior: as within-cluster autocorrelation increased, residual error and the resulting confidence intervals widened as expected.

A key caveat is that clustered standard errors rely on asymptotic theory—they converge to the true standard error only as the number of clusters grows. With too few clusters, inference is unreliable. In our simulations, false positive rates were elevated at small cluster counts but settled into the expected range as the number of units increased.

Permutation Testing

Rather than modeling the correlation structure, permutation testing sidesteps it entirely by constructing a null distribution from the data as observed. The procedure begins with a standard DID estimate. That estimated treatment effect is then removed from the treated units in the post-intervention period. Next, the treatment and control labels are randomly shuffled, and the DID model is refit on the scrambled assignment. Repeating this thousands of times yields a sampling distribution of effects that could have arisen had the intervention had no real impact. The observed effect's position within that distribution gives the p-value.

Testing this approach on simulated data with no true causal effect produced false positive rates matching the chosen alpha across most sample sizes.

With very few units, permutation testing never declared significance—not because it was correctly conservative, but because the number of possible label permutations was too small to construct a meaningful null distribution. Averaging, by contrast, remains analytically valid at very small scales, though it eventually fails when variance cannot be estimated at all.

A Practical Comparison

Across our simulations, all three methods converged to the expected false positive rate with enough units, but their behavior diverged at smaller scales. With one to five units, clustering produced high false positive rates, while averaging and permutation appeared conservative—though in the permutation case this was an artifact of insufficient permutation resolution rather than statistical virtue.

At five to ten units, averaging and permutation outperformed clustering, which continued to declare significance too liberally. While clustering is the easiest to implement in standard statistical software, it can mislead on small datasets.

Statistical power told a different story. All methods showed substantial power losses relative to the naive approach, which over-declares everything. Averaging suffered the largest power drop, consistent with its reduction of the dataset. Clustering and permutation performed comparably at larger scales, but clustering's apparent power advantage at small scales is misleading—it catches true positives and false positives with equal enthusiasm.

Our results support a clear set of recommendations:

• Large numbers of units: Use clustered standard errors. Averaging is a reasonable alternative when power is not a concern. Permutation testing is less appealing at this scale due to computational cost.
• Small numbers of units: Choose permutation testing to preserve power while keeping false positives in check. Consider running multiple methods and weighing their error trade-offs against the decision at hand.
• Very few units: All methods become unreliable. Averaging degrades most gracefully, but at a serious power cost. At this point, the better investment is gathering more data.

Conclusions

Where A/B testing is infeasible, DID models offer a path to causal understanding from observational time series data—but only if significance testing accounts for autocorrelation. The naive standard errors from conventional regression output are simply wrong for this data structure.

Our simulations confirm that averaging, clustered standard errors, and permutation testing all correct false positive rates relative to the naive approach. The right choice depends on the data at hand: permutation suits smaller samples, clustering suits larger ones, and averaging serves those willing to sacrifice power for simplicity. The largest takeaway is that autocorrelation cannot be ignored in time series inference, and the method of correction should be chosen deliberately to match the sample size and decision context.