HeadlinesBriefing favicon HeadlinesBriefing.com

LLMs battle Russian propaganda in Estonian benchmark

Ars Technica •
×

The Estonian government released a benchmark that pits dozens of large language models against Russia’s “strategic narratives.” Open‑weight models such as Nvidia’s Nemotron and Alibaba’s Qwen performed on par with Anthropic’s top offerings, while OpenAI’s GPT‑5.4 earned an “Exemplary” rating on 54 percent of prompts and posted an 88.9 mean score. These scores help policymakers gauge AI’s role in information warfare.

Newer frontier models showed markedly higher resistance than legacy systems. Claude 3.5 Haiku, the highest‑rated 2024 release, scored only 73.1, placing it in the bottom third of 2026 models. Google’s Gemini 2.5 Pro achieved a mean of 82 but faltered on prompts, and its successor Gemini 3.5 Flash dropped to 73, matching Anthropic models from two years earlier. Google attributes the dip to prompt‑injection testing during development.

The benchmark also revealed language‑specific gaps: several models, including Gemini 3.5 Flash, Moonshot’s Kimi K2 and StepFun’s Step 3.5 Flash, performed worse on Russian‑language queries. Researchers warn that Russia’s push to embed culturally tailored narratives into AI, bolstered by BRICS alliances, could amplify misinformation if developers do not prioritize multilingual robustness. Enterprises must audit outputs in target markets.