Measuring agents' ability to improve and generalize on open ended ML research tasks.
Progress toward recursive self-improvement is predicated on the development of agents that can solve hard, open ended ML research problems over long periods of time. Real open-ended research relies on iteration - often featuring an evolving and improving solution driven by reactions to feedback and the exploration of a diversity of approaches.
One-shot and few-shot evaluations [1] miss out on the opportunity to measure this evolution. An agent that starts behind another but makes good use of feedback may eventually outperform one that improves more slowly. Standard auto-research evals [2] and other long-horizon benchmarks [3] may not adequately distinguish between progress on test scores and overfitting on validation scores.
To address this gap, we introduce AutoResearchExam, a benchmark of open ended ML research tasks designed to measure how quickly agents can improve a private test score over 24-hour auto-research.
We detail three contributions:
This section presents the headline results. We examine research strategies and model behavior in Analyzing agent behavior.
Research area
We run AutoResearch with the standard Terminus 2 harness in a 24-hour benchmark window. Harnesses can significantly affect performance, so we compare all models using the same Terminus 2 harness to isolate model performance and evaluate them fairly. The harness itself lets each agent submit as many times as it wants. After every submission, the agent receives a validation score and its remaining time in the 24-hour window. It never sees hidden test scores. See the harness comparison for more information about the harness we used, and our comparison with Claude Code and Codex.
Evaluating agents over extended 24-hour runs lets us measure whether models sustain progress and whether repeated optimization of validation scores leads to overfitting, with gains that fail to generalize to hidden test data.
Each task uses the Harbor format, with separate work and verifier containers. To guard against reward hacking, our adversarial quality control and reward hacking agent inspects task environments and completed model rollouts. We have already used this agent to find issues in several widely used benchmarks and are working with the community to address them (blog coming soon). Models also have no internet access.
We ran each model at the highest effort level available when we generated its rollouts. For Muse Spark 1.3, this was xhigh; its current highest level, Max, was not yet available. Each model can produce at most 32,000 output tokens per turn.
In each task, the agent is asked to optimize a specific metric, such as accuracy, speedup, perplexity, or mean squared error. This metric is a score which cannot be compared directly across tasks. For instance, Opus 5 achieves 70.67% worst-group accuracy on "Selecting Bird Images Across Hidden Background Groups", and 41.28 dB mean PSNR on "Accelerating video diffusion by reusing predictions". Pass/fail tasks can use average pass rate, but open-ended tasks need a common reward scale.
We use baseline anchor scores and reward maps to convert a raw score into a final reward which is given to the agent. In particular, we distinguish between the raw "score" for the metric achieved by the agent and the resulting "reward". Note that the reward is what is optimized by the agent, and tracked in the AUARC. A trivial baseline receives reward 0. Some maps reserve the range from 0 to 0.1 for weaker trivial baselines. A strong reference solution typically receives reward 0.5.
Depending on the natural raw score range for a task, we use three maps: bounded scores in [a, b], signed unbounded scores in (−∞, ∞), and unsigned unbounded scores in (0, ∞). For bounded metrics, the map gives more credit to the same absolute improvement near the known best score, where progress is usually harder. For example, improving accuracy from 0.85 to 0.95 can earn more reward than improving it from 0.10 to 0.20.
The sketches below illustrate our mapping from raw score to [0, 1]. Such a mapping lets us account for the changing difficulty of making marginal improvements at different score ranges, and lets us average scores in a meaningful way. We illustrate this for metrics where a higher score is better. We appropriately flip the map when a lower score is better.
(−∞, ∞)→[0, 1)
We use this map when there is no fixed best possible score.
Examples include episode return, log likelihood, and objective values without a fixed useful range.
| reward | Returned score. |
|---|---|
| rewardbaseline | Reward at the baseline score. |
| u | Progress in units of the reference improvement. |
| s | Raw score. |
| s0 | Baseline raw score. |
| sref | Reference raw score. |
| σ | Direction sign. It is +1 when higher scores are better and −1 when lower scores are better. |
(0, ∞)→[0, 1)
We use this map for positive metrics where improvements are meaningful in ratios and may span many orders of magnitude.
Examples include throughput, speedup, and inverse error.
| reward | Returned score. |
|---|---|
| rewardbaseline | Reward at the baseline score. |
| u | Progress in units of the reference improvement. |
| s | Raw score. |
| s0 | Baseline raw score. |
| sref | Reference raw score. |
| σ | Direction sign. It is +1 when higher scores are better and −1 when lower scores are better. |
[s0, s1]→[0, 1]
We use this map when a metric has a known best value.
Examples include accuracy, F1, and AUC with a best value of 1, plus error, latency, and regret with a best value of 0.
| reward | Returned score. |
|---|---|
| rewardbaseline | Reward at the baseline score. |
| u | Progress in units of the reference improvement. |
| s | Raw score. |
| s0 | Baseline raw score. |
| s1 | Best possible raw score. |
| d0 | Distance from s1 at the baseline score. |
| d | Distance from s1 at the agent score. |
| dref | Distance from s1 at the reference score. |
To motivate our main evaluation metric, we begin by considering the validation and hidden test score plots below. These plots show the performance of Claude Fable 5.1 working on Faster language model generation on CPUs. The left curve plots the best submission up to time step t, as selected by the best observed validation score. The right curve plots the test score of the submission with the best validation score so far. The agent never sees the hidden test scores nor the curve on the right. Note that the hidden test curve is not necessarily monotonic; if the agent is overfitting on the validation set, the hidden test score may experience a drop relative to a prior submission (indeed, this is observed in the below plot).
We would like to design a metric which captures both the generalization performance of any solution found by the model, whilst simultaneously rewarding early improvements. We argue that the Area Under the AutoResearch Curve (AUARC) provides a suitable measure. This metric is equivalent to the time weighted average hidden test score. Formally, it is defined simply by:
Here, ri is the hidden test reward held during interval i, Δti is the length of that interval, and T is the total run time. A score held for more time contributes more area. An improvement made early therefore adds more than the same improvement made near the end.
In this section, we examine the set of 24-hour rollouts we collect in order to obtain insights into the approaches taken by different models.
Our first question is: Do models continue improving over longer runs? To investigate this, we construct the following two plots. The first, below on the left, plots the average hidden-test set reward at each hour interval, for each model. This plot demonstrates that agents continue improving throughout the 24h time period, following a similar pattern to the results from EdgeBench on long-horizon tasks [3]. The second plot, on the right, shows what fraction of results (on the y-axis) result in improvements on the test-set across time (x-axis).
These plots show that Astra often starts with a strong solution. It has the higher fitted hidden-test reward for about the first three hours, after which Fable 5.1 then begins to outperform on average. By hour 22, Fable 5.1 narrowly leads in the hidden-test area under the curve (AUARC), the metric we use to evaluate overall performance on our leaderboard.
Second, the right plot shows that Muse has the largest share of runs that still improve after the 12-hour mark, at about 79%. Qwen and Grok follow at about 76%. This means that the agents are still working and improving on their solutions within the environment, that is, that the benchmark itself supports extended amounts of experimentation or optimization.
We compare validation and hidden-test AUARC. A smaller gap means closer agreement between the feedback agents optimize and their performance on unseen data.
For each model, we average validation AUARC (V) and hidden-test AUARC (T) across its included tasks. We then calculate the relative gap:
A larger positive gap means less of the validation performance carries over to hidden tests. Lines show 95% intervals from resampling tasks. The gap alone does not prove overfitting.
We use a prompted model to review each sampled round and classify it as HPO-only when the agent only tunes settings of an existing method, without introducing a new method or substantive fix.
In our sample, Anthropic's Fable and Opus spend fewer rounds doing only HPO than other models. Both sometimes test several variants using local validation before submitting, suggesting they may shift some experimentation from benchmark submissions to internal checks. This points to a difference in research approach, though it does not establish that doing less HPO causes their stronger results. Another interesting point of note is that Sol has a very large number of submissions which are considered HPO by our parser; we believe that this is most likely due to the fact that Sol makes a large number of submissions overall (see Do models that submit more perform better?), and chooses to do less internal tuning on a dev set prior to submission.
We see only a weak association between submission count and test AUARC across models. We don’t see evidence that simply submitting more leads to higher scores on this benchmark. This suggests that naïve hill climbing through more submissions alone is insufficient, and that the quality of research steps matters.
We condense insights from a set of AutoResearch trajectories into a single sentence and use it as a hint for a 200-turn Terminus 2 run. We compare its hidden-test reward with the same model’s first 200-turn AutoResearch iteration without a hint.
Average across 22 CPU tasks.
| Run |  Claude Opus 5 |  Muse Spark 1.3 |
|---|---|---|
| Terminus 2 · 200 turns | ||
| Without hint | 0.441 | 0.403 |
| With validation hint | 0.547 | 0.447 |
| Full AutoResearch | 0.592 | 0.449 |
Shaded segment: gain with a hint.
The result without a hint is from the first AutoResearch iteration. Full AutoResearch shows the final reward after the longer run, not another 200-turn result.
The hint moves Opus much farther on average than Muse, and gets Muse close to its own final AutoResearch score. This suggests that insights discovered during AutoResearch can help models reach better solutions within a single 200-turn run.
First, we give some additional details on our Terminus 2 autoresearch harness:
A natural question is whether our Terminus 2 based autoresearch harness will have significantly different results when compared Codex and Claude Code. To evaluate this, we compared all three harnesses with Opus 5 and GPT-5.6 Sol on a subset of on five tasks over 12 hours.
Although we found that the average rewards differ slightly across the harnesses, the aggregate hidden test ranking between models stays the same. These results support our choice to use Terminus 2 for all models. It provides a consistent setup and preserves the aggregate model ranking seen with native harnesses in this comparison. See the below plots for the individual run information.
If you use AutoResearchExam in your research, please cite:
@misc{ramesh2026autoresearchexam,
author = {Ramesh, Anirudha and
Devic, Siddartha and
Garg, Shivank and
Parulekar, Advait and
Mahnot, Drish and
Pimpalgaonkar, Shreyas and
Suresh, Vishnu and
Dimakis, Alex and
Sathiamoorthy, Maheswaran},
title = {{AutoResearchExam}},
year = {2026},
month = sep,
url = {https://benchmarks.bespokelabs.ai/autoresearchexam/},
note = {Published September 9, 2026}
}