In product analytics, treatments are user‑chosen (features, communities, free trials). Without randomization, analysts rely on quasi‑experimental designs. Two popular approaches are Lagged‑outcome regression—often estimated with double machine learning (Double ML) and rooted in Causal Inference for the Brave and True—and Difference‑in‑Differences (DiD)—popularized by The Mixtape and updated by Sant'Anna and Zhao (2020), also implemented in Double ML.
The key distinction lies in their identifying assumptions. Lagged‑outcome regression assumes unconfoundedness: after conditioning on past outcomes and covariates, no unobserved confounding remains. DiD assumes parallel trends: absent treatment, the treated and control groups would have followed the same trajectory. The article’s simulation study shows when each assumption breaks, giving a practical rule for choosing the right design.
In practice, Lagged‑outcome regression compares users with matched pre‑treatment histories and attributes, attributing post‑period differences to the treatment. DiD allows different starting levels but treats the pre‑period gap as permanent, estimating the change in that gap after treatment. Both can be estimated with regression adjustment, inverse propensity weighting, or doubly robust methods, yet each design is biased if the underlying gap assumption does not hold.
Source: Towards Data Science · Summarized by HeadlinesBriefing