A refreshed version of StarSkirmish Bench, a benchmark in which large language models play StarCraft: Brood War by writing code, has been released with results from the flagship models of OpenAI, Anthropic, Google, and Meta. Star Skirmish is a benchmark where LLMs write bots for StarCraft: Brood War. Each model gets one hour of wall clock time to write a Protoss bot in C++. That bot is then evaluated against other LLMs' bots and established human-written bots. The benchmark is designed to test long-horizon reasoning and agentic coding, since models must incorporate lessons from practice games to improve their bot.
In the v0.2 update, GPT-6 Astra is the top-scoring model, a clear step ahead of GPT-6.1 Sol in second place. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind them, all scoring between 37 and 39. Gemini 4 Argon runs at the highest reasoning effort available to it, while every other model runs at max. Scores are average win rates from each model's five runs against a roster of competitive human-written and demo bots.
Cost is also considered. GPT-6.1 Sol offers unusually good value: it is the only model in the most attractive quadrant, scoring second at just over a tenth of GPT-6 Astra's cost per run. Gemini 4 Argon varies far more between runs than other flagships, with its best and worst runs 18 points apart.
The strongest human-written bots remain well ahead of every model. Against Stardust, Purple Wave and Banana Brain, the best model, GPT-6 Astra, wins only 7% of its games. The developers note that one-hour reasoning limits are a factor, but real tournament conditions would be significantly harder.
Source: Hacker News · Summarized by HeadlinesBriefing