Traditional offline evaluation for recommendation systems treats user interactions as static historical data, ignoring how recommendations actively shape user behavior. This observational approach fails to capture the true impact of new recommenders, leading to flawed metrics that don't reflect real-world outcomes.

Counterfactual evaluation addresses this by estimating what would have happened under a different recommendation policy, using techniques like Inverse Propensity Scoring (IPS). This method reweights logged data based on how often a new model would recommend items versus the existing production system, providing a more realistic performance estimate.

Key challenges include insufficient support when new models recommend items never shown by the old system, and high variance from large probability differences. Solutions like Clipped IPS (CIPS) and Self-Normalized IPS (SNIPS) help stabilize estimates. While more complex than A/B testing, this approach is crucial for research and safe model deployment.