#1 OpenAI, o3 announcement (as reported by TechCrunch) said on 2024-12-20 · source
o3 could answer just over a fourth of the questions on FrontierMath.
OpenAI did announce a score of over 25% for o3 on FrontierMath, but Epoch AI's independent test of the publicly released model showed a significantly lower score of around 10%. The discrepancy suggests that the initial claim by OpenAI might have been based on an internal version with different settings. Additionally, Epoch AI, the creator of FrontierMath, was funded by OpenAI, which raises questions about the independence of the benchmark. These points lead to the conclusion that the claim is partially contradicted.
Citations
- techcrunch.com: “"We're seeing [internally], with o3 in aggressive test-time compute settings, we're able to get over 25%."”
- techcrunch.com: “Epoch AI, the research institute behind FrontierMath, released results of its independent benchmark tests of o3 on Friday. Epoch found that o3 scored around 10%, well below OpenAI's highest claimed score.”
- techcrunch.com: “Epoch AI, a nonprofit primarily funded by Open Philanthropy, a research and grantmaking foundation, revealed on December 20 that OpenAI had supported the creation of FrontierMath.”
What would change this
- Direct evidence from OpenAI confirming the exact configuration and settings used for the internal version of o3 that achieved the 25% score.
- A detailed explanation from Epoch AI on how the differences in the testing environment (such as compute, scaffold, and problem set) contributed to the lower score in their independent test.
Earlier versions (1)
- Partially contradicted v1 · qwen-max · 2026-09-22 15:57
The evidence supports that OpenAI did announce a score of over 25% for o3 on FrontierMath. However, Epoch AI's independent test of the publicly released model showed a significantly lower score of around 10%. This discrepancy suggests that the initial claim may have been based on an internal version with different settings. Additionally, Epoch AI's funding by OpenAI and its access to the problems raise questions about the independence of the benchmark. These points lead to the conclusion that the claim is partially contradicted.
Evidence on this claim
Epoch AI, the research institute behind FrontierMath, released results of its independent benchmark tests of o3 on Friday. Epoch found that o3 scored around 10%, well below OpenAI’s highest claimed score.
by WeirdTrue · 2026-09-23 · excerpt found in source
“We’re seeing [internally], with o3 in aggressive test-time compute settings, we’re able to get over 25%.” As it turns out, that figure was likely an upper bound, achieved by a version of o3 with more computing behind it than the model OpenAI publicly launched last week.
by WeirdTrue · 2026-09-23 · excerpt found in source
“The difference between our results and OpenAI’s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time [computing], or because those results were run on a different subset of FrontierMath (the 180 problems in frontiermath-2024-11-26 vs the 290 problems in frontiermath-2025-02-28-private),” wrote Epoch.
by WeirdTrue · 2026-09-23 · excerpt found in source
Epoch AI, a nonprofit primarily funded by Open Philanthropy, a research and grantmaking foundation, revealed on December 20 that OpenAI had supported the creation of FrontierMath.
by WeirdTrue · 2026-09-23 · excerpt found in source
has scored a breakthrough 75.7% on the Semi-Private Evaluation set at our stated public leaderboard $10k compute limit. A high-compute (172x) o3 configuration scored 87.5%
by WeirdTrue · 2026-09-23 · excerpt found in source
OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.
by WeirdTrue · 2026-09-23 · excerpt found in source