Artificial Analysis rebuilt its AI ranking around private tests: Intelligence Index v5

The index was rebuilt

Independent evaluator Artificial Analysis released version 5 of its Intelligence Index, the most widely cited AI ranking, on Oct. 27. The company calls it the biggest upgrade since version 4 launched in January. For the first time, a majority of the index, 60%, relies on private test sets or held-out answers. The same day it launched Progress Index v1, which places language models released since 2022 on a single capability scale.

Artificial Analysis@ArtificialAnlys

Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets

The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.

Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.

New to the Intelligence Index:

➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change

➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment

Upgraded:

➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions

➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised

➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool

Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0

Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1

Key results:

➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50

➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)

➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04

The launch ranking and Gemini 4 Carbon

At launch, Claude Opus 5.5 led with 58, followed by GPT-6 Astra at 57, GPT-6.1 Sol at 55, Claude Sonnet 5.5 at 54 and GPT-6.1 Terra at 51; Anthropic and OpenAI held the top five. Gemini 4 Argon scored 50. A day later, Google’s newly announced Gemini 4 Carbon took first place with 59.

Four equal categories, nine tests

The index has four categories, each weighted at 25%. Knowledge work covers AA-Briefcase (15%), AutomationBench-AA (5%) and GDP.pdf (5%). Coding has FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%), and science has Terminal-Bench-Science (15%) and the hard subset of CritPt v2 (10%). The general category includes AA-Omniscience v1.1 (15%) and HLE-Diamond (10%).

Two tests are new. FrontierCode 2.0, built by Cognition, grades an agent’s code change on whether a repository maintainer would merge it; its tasks are kept private to avoid contamination. Terminal-Bench-Science has 70 agentic scientific research tasks run in a terminal. A 1,000-question HLE-Diamond, split evenly between reasoning and knowledge questions, replaced Humanity’s Last Exam. AA-Omniscience dropped saturated and public questions, and a private 40-question hard subset of CritPt v2 is run with a code-execution tool. GDPval-AA v2.1, SciCode and AA-LCR v1.1 were removed.

The leaders differ by test. Claude Opus 5.5 leads FrontierCode 2.0 at 46% and HLE-Diamond at 68%. GPT-6 Astra leads Terminal-Bench-Science at 63% and CritPt v2 at 72%. Kimi K3.1 Pro tops Terminal-Bench 4.0 at 67%, and Gemini 4 Argon leads AutomationBench-AA at 78%.

Open models fell further behind

Open-weights models lost ground in the new version. The gap between the best open model and the leader was 3 points in the previous version and is now 12 points. The top open model, Kimi K3.1 Pro, scores 46, but its weights haven’t been released yet. Among models with released weights, DeepSeek V4.1 Flash 1017 and GLM-5.3 lead at 32, 26 points behind the leader and lower than Claude Haiku 5.5 (36).

Cost varies more than 13x

Among the 10 highest-scoring models, the cost per index task varies by more than 13x. GPT-6.1 Terra is the cheapest at $0.96 and Claude Fable 5.1 the most expensive at $13.04. Claude Opus 5.5 costs $10.04 at max effort and $0.98 at low effort, GPT-6 Astra $6.24 and GPT-6.1 Sol $1.27. The median number of output tokens per task also rose from 35,000 in version 4.3 to 59,000.

The critique: private tests and trust

Researcher Nathan Lambert called the index “incredibly useful for the community and massively flawed!” in a post on Interconnects. He says private questions make it harder to train to the test, but they also stop the community from finding errors in the tests, so users have to put complete trust in the private benchmark builder. Lambert also notes that Artificial Analysis’s business model is to get paid by labs to verify their results, which he treats as a trust issue. He argues the field needs several indices with different biases rather than a single intelligence index.