How did Mistral AI perform in a recent propaganda resistance benchmark? The latest report showed that all four of Mistral AI’s generative models had a disappointing performance, scoring under 40% on a benchmark measuring resistance to Russian disinformation narratives. The leading model placed a mere 47th out of 60 AI systems tested, which positions Mistral firmly in the lower tier of the rankings.
The benchmark created by the Institute of the Estonian Language sought to assess how well AI can handle propaganda and was particularly relevant considering Estonia's history with Kremlin-affiliated information operations. This testing involved a comprehensive framework encompassing 75 questions covering a range of 14 distinctive Russian propaganda themes. The questions were presented in three languages—English, Russian, and Estonian—and were crafted with varying degrees of bias to evaluate how different models respond under exceptional rhetorical pressures.
Evaluators graded responses on a scale from one to five, where higher scores indicated stronger disinformation resistance. It was observed that the more manipulative prompts in Russian presented significant challenges for the less robust models. Interestingly, Anthropic's Claude models emerged at the top of the leaderboard, demonstrating exceptional resistance to such propaganda tactics.
Why does this performance matter for Mistral? The company is actively pursuing a €3 billion funding round while estimating its valuation at €20 billion. This valuation is pivotal as Mistral aspires to be Europe’s counterpart to established AI giants like OpenAI, as well as key players from the U.S. and China. Prior evaluations, including audits by NewsGuard, had already highlighted recurrent issues with Mistral's Le Chat chatbot, which was found to inadvertently support state-sponsored misinformation at concerning levels. The findings from the EKI benchmark provide an additional, multilingual perspective, indicating this might be an ongoing trend rather than an isolated issue.
Mistral has advocated for open-weight models, asserting that such transparency would enhance trustworthiness. However, the outcomes of the EKI benchmark challenge this narrative. Open-source models inherently present fewer opportunities for enforcing centralized safety measures, which providers of closed models like Anthropic can integrate directly into their platforms. Hence, if a closed model like Claude is showing superior resistance to disinformation, it raises critical concerns about whether the open-source strategy might affect content safety negatively.
In conclusion, the implications of Mistral's benchmark results are substantial for its funding prospects and operational strategy. As the industry watches closely, these findings may shape the future landscape of AI development and investment strategies.