HeadlinesBriefing favicon HeadlinesBriefing.com

Testing auto.arima Stability with Rolling Validation

Towards Data Science •
×

Econometric models face a fundamental tension: accuracy measures what a model learns, while stability measures how it learns. As variable counts explode in time series forecasting, distinguishing between these becomes critical. The article argues that traditional criteria like AIC optimize only for information retention, not structural robustness — a model can appear accurate while its coefficient estimates swing wildly with each new observation.

The author tests this using R's forecast package and its auto.arima function on a simulated AR(4) process with known coefficients {0.7, -0.2, 0.5, -0.8} across 1,000 periods. Because temporal dependencies forbid random subsetting, rolling validation replaces k-fold cross-validation. Training begins at n=20 and expands sequentially, tracking coefficient convergence at each step.

Results reveal auto.arima requires roughly 400 observations to stabilize on the true lag structure. Strikingly, out-of-sample forecast accuracy remains comparable even during the first 200 volatile iterations — accuracy masks instability. The author then injects random shocks from a different distribution to stress-test robustness, finding that stable models resist overfitting to anomalies rather than chasing them.

This exposes a dangerous blind spot: practitioners may deploy seemingly accurate models that collapse under minor distributional shifts. Rolling validation of coefficient paths, not just forecast errors, should become standard practice before trusting any autoregressive structure in production.