#1 Meta, Llama 4 launch announcement said on 2025-04-05 · source
Llama 4 Maverick achieves an ELO of 1417 on LMArena, ranking #2.
The claim that Llama 4 Maverick achieved an ELO of 1417 and ranked #2 on LMArena is partially contradicted. While Meta did claim this in their launch announcement, the evidence indicates that the version tested and ranked was an unreleased experimental variant, not the public release. This discrepancy suggests that the claim, as stated, is not fully supported by the evidence. The domain constraint pack also mentions that there is a known pattern of official numbers and independent retests having systematic differences, which aligns with the current case.
Citations
- ai.meta.com: “Llama 4 Maverick offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena”
- www.theregister.com: “According to LMArena on Monday, Meta provided a version of Llama 4 that is not publicly available and was seemingly specifically designed to charm those human voters, potentially giving it an edge in the rankings over publicly available competitors.”
- www.theregister.com: “Meta should have made it clearer that Llama-4-Maverick-03-26-Experimental was a customized model to optimize for human preference,”
- arxiv.org: “We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results. At an extreme, we identify 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release.”
What would change this
- Independent verification of the public version's performance on LMArena after its release.
Earlier versions (1)
- Partially contradicted v1 · qwen-max · 2026-09-22 15:56
The claim that Llama 4 Maverick ranked #2 on LMArena with an ELO of 1417 is partially contradicted. While Meta did make this claim in their launch announcement, evidence shows that the version tested and ranked on LMArena was an experimental variant, not the public release. The public version, when re-tested, ranked 32nd. This discrepancy indicates that the claim, as stated, is not fully supported by the evidence.
Evidence on this claim
- https://arxiv.org/abs/2504.20879cited in v2
We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results. At an extreme, we identify 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release.
by WeirdTrue · 2026-09-23 · excerpt found in source
According to LMArena on Monday, Meta provided a version of Llama 4 that is not publicly available and was seemingly specifically designed to charm those human voters, potentially giving it an edge in the rankings over publicly available competitors.
by WeirdTrue · 2026-09-23 · excerpt found in source
"Meta should have made it clearer that Llama-4-Maverick-03-26-Experimental was a customized model to optimize for human preference," the group added.
by WeirdTrue · 2026-09-23 · excerpt found in source
we will mark model scores as "provisional" until additional 2,000 fresh votes have been collected after model release, if more than 10 models were pre-release tested in parallel.
by WeirdTrue · 2026-09-23 · excerpt found in source
Llama 4 Maverick offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena
by WeirdTrue · 2026-09-23 · excerpt found in source