Difference-in-Differences

What is difference-in-differences?

Difference-in-differences is a quasi-experimental design for situations where random assignment is unavailable or an intervention is delivered to an existing group. It does not ask whether the treated group has a higher outcome than the comparison group after launch. Such a comparison can be misleading because groups may already differ. Instead, DiD subtracts the comparison group’s before-to-after change from the treated group’s before-to-after change. The remaining difference is attributed to the intervention only under explicit assumptions.

A retailer may introduce a new fulfillment policy in one region while another region continues operating normally. Revenue, delivery times, and customer mix can differ across regions even before the policy. If treated-region delivery time falls by 1.8 days while comparison-region delivery time falls by 0.5 days over the same dates, the simple DiD estimate is minus 1.3 days. The comparison region is intended to represent the change the treated region would have experienced without the policy.

DiD is useful for geo rollouts, policy changes, operational launches, and product changes that cannot be randomized at the user level. It is not a weaker spelling of an A/B test. Randomization makes treatment and control exchangeable at baseline; DiD instead relies on a defensible counterfactual trend. That makes design choices, pre-period diagnostics, and honest uncertainty especially important.

The basic design and formula

Let Y_T,post and Y_T,pre be average outcomes for the treated group after and before launch. Let the corresponding comparison outcomes be Y_C,post and Y_C,pre. The estimate is:

DiD = (Y_T,post − Y_T,pre) − (Y_C,post − Y_C,pre).

In a two-group, two-period setting, an equivalent regression is Y_it = α + β Treated_i + γ Post_t + δ(Treated_i × Post_t) + ε_it. The interaction coefficient δ is the DiD estimate. The treated indicator absorbs stable level differences, and the post indicator absorbs shocks common to both groups. With many locations and dates, analysts commonly use group and time fixed effects, but a familiar regression does not automatically make its causal interpretation valid.

Choose the outcome scale before analysis. A difference in conversion-rate percentage points answers a different question from a log-revenue model or a relative-change estimate. Define the unit—user, account, store, or region—and aggregate or model at a level consistent with treatment delivery and dependence. If a policy is assigned by region, treating every customer transaction as independently assigned usually produces falsely narrow intervals.

Assumptions and validity conditions

The central identifying assumption is parallel trends: absent treatment, the treated and comparison groups would have changed by the same amount on the selected outcome scale over the post period. The assumption does not require equal outcome levels. It does require that differences in underlying trends are not driving the estimate. Historical outcome charts can make the claim more or less credible, but cannot prove the unobserved counterfactual.

Valid application also requires no treated-group-specific concurrent change that is omitted from the comparison. A marketing campaign only in the treated geography, a supply outage in one comparison geography, a price change, or a different measurement release can break the design. Treatment timing must be accurately recorded. The treatment should not change which units enter the denominator in a way that changes the estimand without being recognized; for example, a delivery promise may alter both purchases and which orders are logged.

Spillovers matter. If comparison-region customers buy from treated-region stores, learn about the new policy, or are affected by reallocated inventory, the comparison group is partially treated. That can bias the estimate toward or away from zero. Anticipation is another threat: behavior can change before the recorded launch date if employees, customers, or merchants know the intervention is coming.

When DiD is appropriate for experimentation

Use DiD when the operational decision genuinely prevents clean randomized assignment but a concurrent comparison group and reliable pre-period data exist. It can evaluate a warehouse process introduced to selected facilities, a legal-policy change affecting one market, or a platform migration completed by one customer cohort first. If users, stores, or accounts can be randomized safely, a controlled experiment is usually preferable because it needs fewer assumptions and produces a clearer causal estimate.

Before adopting DiD, ask whether rollout order is related to expected performance. A team may launch first in regions with strong managers, low complaint rates, or high demand. That selection is precisely what DiD may fail to remove if those characteristics cause different trends. A deliberately randomized phased rollout, a switchback experiment, or a matched pilot can often improve the design.

Worked scenario: delivery promise in selected cities

A marketplace changes its checkout delivery promise from “three to five days” to a calculated arrival date in four pilot cities on April 1. The decision metric is completed orders per eligible checkout session; cancellations and late-delivery complaints are guardrails. Four comparable cities keep the old promise. The team collects daily city-level outcomes for 16 weeks before and eight weeks after launch, excluding a known holiday period under rules documented before analysis.

Before launch, weekly pre-period conversion trends move similarly, although treated cities have a consistently higher level. After launch, treated cities increase by 0.90 percentage points and comparison cities increase by 0.25 points, yielding a DiD estimate of 0.65 points. The team fits a city-by-week model with city and week fixed effects and uncertainty clustered by city, then checks the result with a preplanned aggregation to weekly city means. It also plots leads and lags around April 1 to inspect whether a relative conversion increase began before launch.

The estimate alone does not settle the rollout. Complaints increase in two pilot cities where carrier capacity is constrained. The decision is to expand only where the fulfillment promise can be met, run a randomized city rollout for remaining markets if feasible, and preserve the diagnostic report. Claiming that the message universally caused a 0.65-point gain would ignore the operation-dependent guardrail result.

Analysis and decision process

1. State the decision, target population, intervention date, outcome definition, and treatment-delivery unit.
2. Select comparison groups based on pre-treatment characteristics and operational knowledge, not their favorable post-launch performance.
3. Lock a sufficiently long pre-period; graph levels and changes and document known shocks, promotions, and measurement changes.
4. Specify the outcome scale, aggregation level, fixed effects or estimator, clustering approach, missing-data rule, and guardrails.
5. Estimate the primary contrast and show period-by-period event-time estimates when multiple dates are available.
6. Run credible robustness checks, such as alternate comparison sets, placebo intervention dates, or outcomes that treatment should not affect.
7. Decide using effect size, interval, guardrails, and assumption strength. Label the result observational when the counterfactual remains uncertain.

Uncertainty must reflect the number of independently treated units. With a handful of cities, ordinary customer-level standard errors are not persuasive. Use methods appropriate to few clusters, consider randomization inference when rollout assignment was randomized, and report sensitivity rather than hiding fragility. For broad guidance on planning effects and precision, see how to calculate sample size.

Limitations and common errors

• Different trends: a treated market on a different growth path can create a false lift or loss.
• Bad controls: adding variables affected by treatment, such as post-launch inventory, can block part of the effect or add bias.
• Staggered timing: simple two-way fixed-effects regressions can misweight effects when cohorts adopt at different times and effects vary.
• Few clusters: apparently precise user-level results can be overconfident when treatment varies by only a few groups.
• Changing composition: acquisition, eligibility, or logging changes can alter the observed population rather than the outcome for a stable population.
• Outcome fishing: choosing the period, comparison group, or outcome after viewing results undermines the confirmatory claim.

Frequently asked questions

Does DiD require identical treated and control groups?

No. Stable baseline differences are allowed. The concern is whether their untreated changes would have been parallel on the chosen outcome scale.

How many pre-periods are enough?

There is no universal number. Use enough history to observe ordinary cycles and major seasonal patterns relevant to the decision; a few noisy weeks rarely support a strong trend claim.

Can we use DiD for a conversion metric?

Yes, if the denominator, eligibility, and aggregation are stable and the model reflects the treatment unit. Report absolute percentage-point effects clearly.

What is a placebo test?

It applies the same analysis to a pre-launch date or an outcome treatment should not affect. A large placebo effect is a warning that the design is capturing unrelated movement.

Is DiD valid if rollout timing is staggered?

It can be, but use estimators designed for staggered adoption and heterogeneous effects. Do not assume a basic pooled fixed-effects coefficient answers the desired question.

Summary

Difference-in-differences compares changes, not just post-launch levels, to estimate an intervention’s effect without individual randomization. Its credibility rests on parallel untreated trends, a clean concurrent comparison, stable measurement, and uncertainty that respects the treatment unit. It is a valuable design for constrained rollouts, but it should not be presented as a substitute for randomization when randomization is feasible.

Sources

• Bertrand, Duflo, and Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?”
• Goodman-Bacon, “Difference-in-Differences with Variation in Treatment Timing”
• Callaway and Sant’Anna, “Difference-in-Differences with Multiple Time Periods”

Related terms

• Multivariate Testing
• Multivariate A/B Test
• Factorial Design
• Full Factorial Design
• Fractional Factorial Design