A new multi-agent environment built on Pokémon Showdown's 2v2 format evaluates how large language models coordinate, communicate, and negotiate actions in real time. Using Qwen3.8-27B models as teammates, the setup tests collaboration under uncertainty with hidden opponent sets and random damage. In a sample battle, one agent correctly identifies a type-matchup error made by its partner before committing to a move, demonstrating dynamic correction and strategic splitting of targets.
Results show that free team chat significantly boosts win rates: teams with unrestricted communication won 64% of battles against a scripted max-damage bot, compared to just 42% when no talking was allowed. Even a single message per turn improved performance to 52%. Notably, a single model controlling both Pokémon achieved only 54%, indicating that the conversational exchange itself—rather than centralized planning—drives the advantage. However, free talk consumed 978k prompt tokens per battle, nearly four times the 259k used in silent matches, highlighting the computational cost of dialogue.
The environment also proves discriminative: stronger Qwen3.8-27B teams defeated weaker Qwen3.5-2B teams 96% of the time and 74% against 9B models, establishing a scalable benchmark. Future work includes isolating the impact of observing a partner’s chosen move and developing a public leaderboard.
Source: Hacker News · Summarized by HeadlinesBriefing