A controlled test evaluated Gemini, DeepSeek, ChatGPT, and Claude on four hidden forecasting traps. First, models correctly avoided shuffling time-series data when predicting retail sales, respecting chronology by holding out recent weeks. This basic test passed universally, suggesting reliance on common tutorials rather than deep reasoning.
The real challenge involved four planted traps mirroring production issues: (1) a 'store_traffic' column nearly perfectly correlated with sales but unavailable at forecast time, (2) delayed sales reporting, (3) promotion effects, and (4) structural breaks. The prompt described columns truthfully without hinting at problems. Each model received identical instructions for forecasting 156 weeks of retail sales (Jan 2023–Dec 2025) with trend, seasonality, promotions, and noise.
All scripts were rerun to verify reported metrics matched actual code output. Only results from executable code counted.
The test reveals how well AI assistants handle real-world forecasting pitfalls beyond textbook leakage warnings, emphasizing that plausible-looking outputs require careful validation.
Source: Towards Data Science · Summarized by HeadlinesBriefing