Researchers explored using text-based Multi-User Dungeons (MUDs), originating in the 1970s, as a method to evaluate Large Language Models (LLMs). This proof of concept, conducted over several months using personal computers and approximately $99 in API credits, focused on scoring LLM behavior across four dimensions within a persistent text world.

A surprising finding emerged regarding the reliability of LLM judges used in the evaluation. When two LLM judges were compared, their agreement on certain dimensions ranged from 22% to 85%. Removing the dimensions reliant on one of these judges caused one frontier model to fall six positions on the leaderboard. The aggregate kappa score indicated noise in the instrument without specifying which models were affected.

The authors emphasize this is a proof of concept, not a validated benchmark, highlighting limitations such as only 50 runs per model and the absence of human raters. However, they believe the divergence observed between LLM judges is a generalizable finding for other judge-based benchmarks. All data, code, and the paper are publicly available under CC BY 4.0 and MIT licenses. Phase 2 is planned to enhance robustness with human baselines, multiple judges, and a larger environment.