Even with step-by-step reasoning, five major AI models insisted on switching doors in the Monty Hall puzzle, even when switching mathematically guarantees a loss. The experiment tested six host variants, from the classic setup to a "Monty Fall" scenario where the host lacks car knowledge. Despite correct reasoning in some cases, models maintained near-33% accuracy and confidently quoted rules before ignoring them.
The study highlights how even advanced LLMs struggle with probabilistic reasoning when host behavior changes, revealing a disconnect between their stated logic and actual probabilistic calculation. Researchers noted models often cited the standard 2/3 switching advantage while applying it to scenarios where sticking is the only winning strategy.
Source: Towards Data Science · Summarized by HeadlinesBriefing