HeadlinesBriefing favicon HeadlinesBriefing.com

Information Theory for Forecast Ensembles

Towards Data Science •
×

Traditional distance metrics like MSE and RMSE can no longer distinguish between optimized forecasting models, leaving econometricians unable to confidently select the best approach for time-series prediction. The author demonstrates this limitation using U.S. inflation data — CPI YoY Growth, PPI YoY Growth, Savings Rate, and Business Inventories MoM Growth — fitted with a VAR model and analyzed through Granger causal networks. Impulse responses confirm directional relationships like the Permanent Income Hypothesis, showing stimulus-driven savings increases flow into demand-pull inflation while producer cost passes through to consumers.

The core problem emerges when comparing three forecasting models: accuracy metrics fail to separate performance between two of them, making ensemble weighting circular. The proposed solution shifts topology entirely — using information theory to measure how much each model captures about the source signal's structure, rather than measuring forecast-to-actual distance. This treats each methodology (ARIMAX, Exponential Smoothing, State Space Models) as an observer of an information source, weighting ensembles by their mutual information with the underlying process.

The framework remains nascent but addresses a fundamental bottleneck: as models converge on similar RMSE scores, the field needs metrics that capture structural understanding rather than point-wise proximity. Information-theoretic weighting could let practitioners combine ARMA and ETS models meaningfully — something AIC/BIC cannot do across model classes.