- Grading Gap: In METR’s expanded benchmark, AI judges gave some proposed tests of AI higher marks than a human scorer, potentially masking weak designs.
- Longer Tasks: METR estimates the human hours of work AI agents had even odds of finishing doubled roughly every 3.6 months across model releases.
- Programming Contests: Leading models outscored most human entrants on problems with clear numerical targets, though a few people typically finished ahead.
METR, a nonprofit that evaluates AI systems, expanded its benchmark on November 20 with longer jobs and research assignments that are harder to grade. AI judges gave proposed tests of AI capabilities higher scores than METR’s human scorer did when the model that designed a test was at least as capable as the one grading it. For researchers who rely on one model to check another’s work, those grades could make a weak design look worth developing.
What Makes an AI Test Useful
METR asked models to design tests that would teach its researchers something about AI capabilities they needed to assess. Models tended to propose optimization problems with automatic scores, even when the nonprofit asked for work with little immediate feedback. Older models often came up with simple tasks. Newer ones could design more challenging problems, but METR often found it unclear what those problems would reveal about the capabilities it wanted to measure. The researchers considered some designs borderline useful and included none in the report.
In a separate assignment, AI judges rated the proposed tests against the same question of usefulness. Their grades tracked METR’s human ratings more closely when the models that wrote the tests were weaker than the judges. On designs from equally capable or stronger models, the AI judges tended to give higher scores than the human did.
Anthropic’s Claude Opus 5.5 came closest to the human ratings overall. When grading designs from OpenAI’s GPT-6 Astra, it scored them about 0.2 points higher on average than the human scorer on a scale from zero to one. A gap that size could change the grading instructions’ recommendation on whether to spend more time developing the test.
What the Time Horizon Measures
METR’s 50% time horizon tracks how much work models can complete at a given chance of success. A two-hour horizon means an AI agent has an estimated 50% chance of completing a task that takes a human expert two hours.
The earlier, coding-heavy benchmark was reaching its ceiling as leading models scored well enough that it had trouble telling them apart. METR added 36 new tasks in its TH2 suite alongside the 222-task Time Horizon 1.1 suite. On 34 new tasks, researchers assigned a human work time to one or more target scores. Successful human trials supplied those times for three tasks. The other 31 relied on estimates informed by records and specialist judgment.
To fit the new curve, METR compared how often agents reached a target score with how long a person would need to reach it. Adding tougher tasks lowered the estimated horizon for a given model. Across releases, the 50% horizon on the two suites combined doubled about every 110 days, close to 105 days on the earlier set. The 95% confidence interval for the new doubling time runs from roughly two to five months.
Programming contests illustrate what agents can do when they have a clear score to improve. Across 15 optimization contests, the three strongest models’ median placements were in the 96th to 98th percentiles among human contestants. Agents had 48 hours on a 28-core machine. Human contestants had about three months, and METR estimates a dedicated entrant might have spent 90 to 180 hours competing. In the median contest, roughly four people beat the strongest models.
What Agents Can Do in a Lab
OpenAI’s researchers hand agents tasks such as checking runs for problems. Between August 1 and October 18, OpenAI estimated hands-off success rates of 79% for monitoring runs for problems and 29% for building software environments where AI systems train using rewards, take tests or carry out tasks. Both figures cover jobs its researchers chose to delegate.
METR’s researchers say automating work with easily checked results could speed up AI research substantially. A researcher can check a warning against the run itself. Deciding which experiment might settle a research question takes a different kind of judgment.
Sayash Kapoor, a computer scientist and coauthor of the CRUX preprint, told MIT Technology Review that picking promising questions and experiments could be a major obstacle to automating AI research. In the August preprint, two agents built around Claude Opus 4.8 each had six days to tackle a different open-ended question from an unpublished computer science paper. Both agents did the engineering without human help. They dropped plausible ideas after experiments that were too small to tell whether the ideas worked. The original authors graded the work as conference reviewers. They found some of the agents’ observations relevant and judged that neither write-up substantially answered its research question.


