Did Xai lie about GROK 3 comparative tests?

Debates on AI comparative tests – and their reporting by AI Labs – spill at the general public.

This week, Openai worker accused Elon Musk’s Ai Company, XAI, publishing comparative results for his or her latest AI model, Grok 3. One of the co -founders of XAI, Igor Babushkin, he insisted that the corporate had the suitable.

The truth lies somewhere in between.

IN Publish on the XAI blogThe company has published a chart showing the outcomes of GROK 3 on Aime 2025, a set of adverse mathematical questions from the recent Invitational mathematical exam. Some experts have Aime validity as a AI reference point. Nevertheless, AIME 2025 and older versions of the test are widely used to look at the mathematical ability of the model.

The XAI chart showed two variants of GROK 3, Grok 3 Reasoning Beta and GroK 3 mini reasoning, beating the very best available OpenAI, O3-Mini-High, on Aime 2025. But OpenAI employees on X quickly noticed that the XAI chart XAI chart. He didn’t consider the AME 2025 O3-Mini-High lead to “Cons@64”.

What is Cons@64, are you able to ask? Well, that is the abbreviation for “Conszeus@64” and principally gives model 64 tries to reply every problem in relation and accepts answers most frequently generated as final answers. As you may imagine, Cons@64 tends to extend the outcomes of the models, and skipping it from the chart may cause one model to surpass one other when it shouldn’t be in point of fact.

GROK 3 Beta and grok 3 mini reasoning for AIME 2025 at “@1”-what implies that the primary result, which models have achieved at a distance-see below the results of the O3-Mini-High. Grok 3 Reasoning Beta also the trail also behind the O1 Openai model on “Medium” Computing. However, XAI is GROK 3 promoting As “the smartest artificial intelligence of the world.”

Babushkin Ox was arguing that OpenAI previously published similarly misleading comparative charts – although charts comparing the performance of its own models. A more neutral event in the talk has developed a more “accurate” chart showing almost every model in Cons@64:

But as a researcher AI Nathan Lambert He identified within the postPerhaps crucial metric stays a secret: the calculation (and money) cost he needed for every model to realize his best result. It simply shows how little a lot of the test tests AI communicates about the restrictions of models – and their strengths.