A novel method and online simulation offer a way to increase statistical power, enabling researchers to achieve more with fewer participants. The core idea involves utilizing a within-subject design where each individual experiences multiple conditions and tasks. Outcomes from these tasks are treated as semi-independent data points, with a statistical correction applied to account for their inherent dependency.

This approach addresses the significant challenges posed by traditional between-subjects designs, which often require an infeasibly large number of participants, especially for studies aiming to detect small effect sizes. The author recounts a "multimillion-dollar government research program" that faced power requirements in the "many thousands" of subjects, a number deemed completely unfeasible.

A practical example illustrates the issue: a simple A/B test comparing a "GUI only" interface to one with an "LLM powered chat guide" might require 126 participants for standard power analysis. The proposed within-subject method, however, allows for more data points from a smaller participant pool. The accompanying online simulation, available at nathanbos.github.io/power-sim, helps researchers estimate the effectiveness of this strategy. The Github repo also provides a guide for the simulation.