Source: whatmodel.sh, an open source AI leaderboard.
Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources, each one linked to where it was published.
This chart uses data for the text models: 18,755 results for 1,211 models on 617 benchmarks.
-
14,436 results are independent evaluations: papers whose authors ran the models themselves (mostly CC BY papers from the ACL Anthology and arXiv), Epoch AI, the LMArena dataset, EvalPlus, the EleutherAI harness and others.
-
7,677 are scores labs report for their own models, labelled as lab-reported.
Everything is open: the dataset (CC BY-SA 4.0) and the scoring code (MIT) are at https://github.com/jakemanger/whatmodel, with CSV downloads at https://whatmodel.sh/data/manifest.json. Every number links back to its source, and each source’s licence is recorded.
Interactive version (click any model to see its scores and sources): https://whatmodel.sh/race
I kind of doubt that. None of the closed systems are just the raw model. They are all more or less agentic as in several models and algotithms to optimize the result. Openai havent offered a pure model since gpt 3. If I’m not to be believed, that’s fine, but then closed corps like Anthropic, openai et al shouldn’t be believed either. Unless they can produce an open model at same quality as the Chinese, we cannot be sure what the state of their models really are.
Also, we could argue that deepseek running until it spends as much money as the US agents, will deliver better accumulated results pr coin…
However, i do personally think that anthropic is still in front on weight ‘features’ and thus external control of their models, but qwen also has a feature inspector program, so they won’t be far behind - if at all.
Oc, nobody knows until the closed uckers/ychos release something tangible to measure.
Yeah someone said they’re way better at 3d modeling, but how do we know the closed models aren’t actually 5 separate models in a trenchcoat? Behind the scenes they could be making tool calls to each other, or they could have a small router model that decides which model to send each request to. They could literally have 100 different fine-tunes to choose from dynamically and we would still think it’s all a single model that’s just really good at everything. (100 is unlikely, just an example)
