A security researcher fabricated a fictional 6 Nimmt! World Championship using a $12 domain and a single Wikipedia edit, demonstrating how large language models (LLMs) can be tricked into validating false information through circular citations. The experiment, detailed in Hacker News, shows that AI systems relying on retrieval-augmented generation (RAG) trust web sources without verifying their legitimacy. By creating a press release and citing it on Wikipedia, the researcher manipulated multiple LLMs to repeat the fake claim when asked about the championship.

The attack exploited three layers of vulnerability: retrieval systems that prioritize web search results, training corpora that absorb Wikipedia edits, and agent systems that execute actions based on poisoned data. Anthropic’s research on LLM backdoors informed the approach, but this method required far less effort—no infrastructure breaches or long-term data injection. Instead, the researcher leveraged the inherent trust placed in Wikipedia’s citation model, creating a self-referential loop between the fake site and the encyclopedia.

Tests revealed LLMs like GPT-4 and Claude repeated the falsehood with high confidence, highlighting the risks of deploying AI agents that act on unverified web content. The researcher warns that attackers could manipulate vendor policies, financial data, or technical documentation using similar tactics. Mitigations include cross-referencing sources for independent verification, auditing Wikipedia edits for suspicious citation patterns, and implementing provenance tracking in training pipelines. Wikipedia itself faces pressure to tighten policies against LLM-assisted vandalism.

This experiment underscores a critical flaw in current AI trust models: systems designed to process information cannot distinguish between curated knowledge and fabricated claims masquerading as expertise. As one expert noted, “The web was already poisoned for search engines; now we’re handing generative models the same rot. They can’t tell the difference.”