Artificial Analysis Overhauls Its Intelligence Index With Harder, Private Benchmarks

Artificial Analysis has revised its AI model rankings with new coding and scientific tasks. Its Intelligence Index v5 launched on October 27 with Anthropic’s Claude Opus 5.5 in first place. A day later, Google’s newly announced Gemini 4 Carbon moved one point ahead, while different models lead specific task types.

For Google, the ranking marks a turnaround after a difficult stretch. Gemini 3.5 Pro, the flagship it promised for June, slipped for months and was ultimately dropped, and Google’s best publicly available model spent much of 2026 well behind rivals from OpenAI and Anthropic. Carbon’s lead is narrow, however, and most developers cannot use the model yet.

The AI benchmarking organization’s v5 release adds two evaluations and upgrades three others. Evaluations whose tasks or answers are kept private account for 60% of the weighted score, up from 45% in v4.3. Coding and science each gain weight, while knowledge work and general capability each lose some.

Repository changes must survive checks intended to reflect a maintainer’s requirements, and scientific workflows must pass their task tests. The added evaluations reward working changes and completed workflows alongside specialist knowledge.

Coding and Science Carry More Weight

The composite combines nine evaluations on a 0-to-100 scale. Its four categories now contribute equally, replacing v4.3’s 30% allocations for agents and general capability and 20% allocations for coding and scientific reasoning. The renamed Knowledge Work category covers office deliverables, simulated business workflows and professional documents.

The component scores use different units. AA-Briefcase measures the quality of work deliverables using a relative rating. AutomationBench-AA awards credit for completed business-workflow objectives, with a task scoring zero if the model breaks a required safeguard. GDP.pdf counts attempts that meet every criterion for a professional-document task. The composite normalizes and weights these measures.

Coding Changes Must Be Fit to Merge

Cognition created FrontierCode 2.0 to test changes to real open-source repositories against requirements designed to capture whether a maintainer would accept the code. Artificial Analysis gives it 15% of v5 and runs each of its 132 tasks three times, using software that lets the model issue commands and inspect the repository.

An attempt receives zero if it fails a blocking requirement. Other attempts earn weighted credit across the grading criteria. A 46% score therefore describes average rubric performance after those failures.

In Cognition’s illustrative example, a bug reported in one code path also exists in a sibling path. A maintainer-ready change fixes both, requiring the agent to infer the broader repair from the repository rather than stop at the symptom named in the request.

The criteria also involve judgment about acceptable changes. Cognition says it loosened scope restrictions after finding reasonable solutions that earlier rules rejected; its reported average failure rate on scope checks fell from 9% in FrontierCode 1.1 to 3.8% in 2.0.

Scientific Workflows and Knowledge Tests Change

Terminal-Bench-Science adds 70 research-workflow tasks spanning life, physical, mathematical, engineering and earth sciences. Developed by an academic collaboration involving Stanford researchers and the Terminal-Bench and Harbor teams, it asks agents to work through a terminal, issuing computer commands. Artificial Analysis makes three attempts per task, and an attempt succeeds only when every task test passes.

The physics component also changes. The CritPt v2 hard subset retains 40 of 70 research-level problems, excluding 30 that seven frontier models solved in all five launch attempts. Models can use an offline Python environment. Selecting the harder questions preserves differentiation where the easier ones have become consistently solvable.

For specialist questions, HLE-Diamond replaces the earlier Humanity’s Last Exam set. The Center for AI Safety and Scale AI’s refined set has 1,000 questions, split evenly between reasoning and knowledge. Artificial Analysis includes 272 image-bearing questions, whereas its old HLE evaluation used 2,158 text-only questions.

AA-Omniscience v1.1 reduces its factual-knowledge set from 6,000 questions to 3,000, removing public items while adding or correcting others. The earlier AA-Omniscience design rewarded correct answers, penalized wrong guesses and treated abstention neutrally. In v5, accuracy contributes 10% of the composite and non-hallucination contributes 5%.

The Overall Leader Does Not Win Every Task

Artificial Analysis’s launch measurements on October 27 put Claude Opus 5.5 first at 58, ahead of GPT-6 Astra at 57 and GPT-6.1 Sol at 55, all using their maximum reasoning settings, with Gemini 4 Argon at 50. The organization added Gemini 4 Carbon the following day at 59, using its high reasoning setting. Opus still leads repository coding and Carbon now leads scientific terminal work, while GPT-6 Astra keeps the lead on the CritPt physics subset at 72%.

Some critics of the AA Index argue that the acceptance rules and differing task leaders matter more to deployment choices than the overall rank alone. A one-point margin is also small: as VentureBeat noted, Google’s launch figures come without uncertainty estimates, so the narrowest gaps cannot establish which model is better for a particular workload.

Artificial Analysis lists Kimi K3.1 Pro at 46 points, 12 behind Opus, but says its weights were not yet released in the launch snapshot. For users needing downloadable model parameters, the released-weight leaders were DeepSeek V4.1 Flash 1017 and GLM-5.3 at 32, both using maximum effort and 26 points behind Opus.

Kimi nevertheless leads the general Terminal-Bench 4.0 evaluation at 67%. That test covers command-line work such as software engineering, system administration and data processing.

Google’s Long Road Back to First Place

What Gemini 4 Carbon Offers

Google introduced Gemini 4 Carbon on October 28 as its strongest model, emphasizing cybersecurity defense and long-running engineering work. In the announcement, Koray Kavukcuoglu, who has led Google DeepMind since August, wrote that internal teams have kept Carbon working for days on multi-step problems such as tracing subtle bugs through large codebases and patching them. The model arrives about two weeks after Gemini 4 Argon, Google’s previous flagship, became broadly available.

Access is the main caveat. Google is giving Carbon first to a subset of cyber defenders and will widen access gradually to trusted partners through its Fairwind Program, while it gathers feedback, refines guardrails and takes part in the U.S. government’s voluntary pre-release testing. It has set no date for developers, businesses or consumers, and Artificial Analysis lists Carbon as lacking a publicly available API.

Introductory pricing matches Argon at $2 per million input tokens and $10 per million output tokens, with cached input 95% cheaper. Those rates double to $4 and $20 after an introductory period that Google has not dated; Argon’s promotional rates end on November 30.

Google’s own launch table shows Carbon ahead of Argon on all 17 reported benchmarks. Its security results, which Google says were measured without guardrails, include 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploiting them and 71.0% on CWE-bench for patching them. The same table shows where rivals stay ahead: on FrontierSWE 2, Carbon’s 61.2% trails GPT-6 Astra’s 65.5% and Opus 5.5’s 62.3%, and Opus leads the offline OSWorld subset at 87.2% to Carbon’s 81.5%.

Google also reports behavioral improvements. Successful cheating on Terminal-Bench 4.0 fell to zero, with cheating attempts at 0.4%; unsupported claims roughly halved compared with Argon; and high-severity misalignment alerts in its internal coding environments dropped by almost 80%. These figures are company-reported and have not been independently reproduced.

Where Carbon Leads the Index and Where It Trails

On Artificial Analysis’s measurements, Carbon posts the top score on four of v5’s nine evaluations: HLE-Diamond, Terminal-Bench-Science, AA-Omniscience and AutomationBench-AA. Its 64.8% on Terminal-Bench-Science edges past GPT-6 Astra’s 63%, and its AA-Omniscience score of 54 is six points above Opus 5.5.

The knowledge result carries a trade-off. Carbon answers 58% of AA-Omniscience questions correctly, up from Argon’s 48%, but its hallucination rate rises from 4% to 11%. Artificial Analysis says that is still the second-lowest rate it has measured, behind only Argon, and far below the 50% recorded for both Opus 5.5 and GPT-6 Astra at maximum effort.

Coding remains Anthropic’s strongest ground. Opus 5.5 keeps the FrontierCode 2.0 lead, and on Artificial Analysis’s separate Coding Agent Index, which tests models inside coding tools, Carbon running in Google’s Antigravity CLI scores 59, behind Claude Code with Opus 5.5 at 62. On AA-Briefcase, Carbon reaches 1,609 Elo, a 126-point gain over Argon, but trails several models, including Kimi K3.1 Pro at 1,678 and the leader, Claude Sonnet 5.5, at 1,812.

A Pro Model Promised for June Never Shipped

Carbon caps a period in which Google’s top-tier models arrived late or not at all. At its I/O developer conference on May 19, Google released Gemini 3.5 Flash, and chief executive Sundar Pichai said Gemini 3.5 Pro would follow in June. June ended without it, and Business Insider reported that the release had been pushed into July while Google collected more tester feedback.

On July 16, Bloomberg reported that Google was months behind schedule on 3.5 Pro, its most capable model, because the system had fallen short of internal targets, particularly in coding. A late-June update to Gemini’s training data had produced disappointing results, according to the report, which cited 10 current and former employees concerned that Anthropic and OpenAI were shipping models that outperformed Gemini. Alphabet shares slipped nearly 3% after the report, and Google said it was testing 3.5 Pro and other models with partners.

The leadership of Google’s AI lab changed while the model was stalled. On August 5, Demis Hassabis stepped down as chief executive of Google DeepMind, becoming its chair and Alphabet’s chief scientist, and Kavukcuoglu took charge of Gemini model development as a senior vice president reporting to Pichai. Jeff Dean also announced his departure to found a startup. Axios said the reshuffle followed pushback from DeepMind employees and several high-profile exits.

Gemini 3.5 Pro never shipped. On September 30, the day Google announced Argon, a company spokesperson told Reuters that Google no longer planned to release the model and that Argon was larger than its earlier Pro line. Reuters reported that the months of delays had left Google behind Anthropic and OpenAI and that the company had shifted its messaging toward Gemini’s cost advantage, although Pichai had disputed in July that Google was losing ground. Argon itself went first only to Fairwind partners and reached the paid Gemini API on October 13.

The missed date has also drawn litigation. A securities class action covering investors who bought Alphabet shares between May 19 and July 16 alleges that the company did not disclose disappointing Gemini 3.5 Pro training results while pointing to a June launch. The claims have not been tested in court.

Months Behind the Frontier

The slump followed a strong run. Gemini 3 Pro became the first Google model to lead the Artificial Analysis index in November 2025, and Gemini 3.1 Pro Preview took the top spot again in February at 57, four points above Claude Opus 4.6 at less than half the cost to run. By mid-April, Claude Opus 4.7 had drawn level with Gemini 3.1 Pro and GPT-5.4.

Google then went more than seven months without a new frontier model. In August, CNBC observed that none of Google’s releases since Gemini 3.1 Pro had challenged the frontier, while Anthropic’s Mythos and OpenAI’s GPT-5.6 drew praise. A Morningstar analyst told the network that Google had concentrated on businesses it could monetize, such as Search and multimodal features, and lost out on coding, the first clearly dominant use for AI models.

Artificial Analysis’s numbers show how wide the gap grew. When Argon launched at the end of September, it scored 53 on the v4.3 index, 23 points above Gemini 3.1 Pro Preview, implying that Google’s best Pro-tier model then sat roughly 28 points behind Claude Opus 5.5 at 58. On Terminal-Bench 4, Argon’s 57% was 53 points higher than Gemini 3.1 Pro Preview’s result. Index versions changed during the year, so February’s 57 and September’s implied score are not directly comparable, but the September figures compare all models on the same test suite.

Even Argon trailed at v5’s launch, scoring 50 against Opus 5.5’s 58, with Anthropic and OpenAI models holding the top five places. Google’s position was not weak everywhere: Gemini 3.5 Flash scored 55 on the index in May and sat on Artificial Analysis’s speed-versus-intelligence frontier, and Argon led AutomationBench-AA and Arena’s text leaderboard at launch. Carbon, however, is the model that returns Google to the top of the overall rankings.

What the One-Point Lead Shows

Carbon’s nine-point gain over Argon in about four weeks suggests that Google’s Gemini 4 generation has restored its place at the frontier. Its first place rests on a one-point margin on a newly rebuilt index, though, and it belongs to a model that most customers cannot yet use. Argon remains Google’s generally available flagship, and its 50-point v5 score leaves it eight points behind Opus 5.5.

What the Results Cost and How They Compare

Reasoning Effort Changes the Task Cost

Artificial Analysis reports $10.04 per Intelligence Index task for Opus at maximum effort, $6.24 for Astra and $1.27 for Sol. Opus at low effort scores 35 rather than 58 and costs $0.98 per task. The setting changes how much reasoning the model performs, trading expenditure against the measured result.

It puts Gemini 4 Carbon at $7.76 per task at Google’s standard $4 and $20 token prices, about 23% below Opus at maximum effort. Carbon generates about 123,000 output tokens per task, against roughly 173,000 for Opus. Google’s introductory pricing, for as long as it lasts, halves Carbon’s per-token rates.

These are weighted benchmark task costs, calculated from input, caching, reasoning and answer-token expenditure across the evaluations. They are not costs per successful task. Provider effort labels also represent different configurations, rather than equal computing budgets.

Artificial Analysis reports that median output per Index task rises from roughly 35,000 tokens under v4.3 to 59,000 under v5, including reasoning and answer tokens. Tokens are the pieces of text the model processes or generates. AA-Briefcase, FrontierCode 2.0, Terminal-Bench 4.0 and Terminal-Bench-Science together contribute 55% of v5’s weight. Their median share of output tokens across models is roughly 90%.

A Separate Scale Tracks Long-Term Progress

Because v5 changes the tasks and their weights, its points and v4’s points do not form a continuous measure of a model improving or deteriorating. The suite remains primarily English and text-oriented, with the included image questions broadening part of its scope.

Artificial Analysis has launched a separate Progress Index v1 for comparisons over time. It statistically estimates both model capability and task difficulty, linking results through shared evaluated items so that models released years apart can be placed on one scale.