Shared taskMass spectrometry
Spectrum to Molecule
Build a program that identifies the compound behind a mass spectrum among 4.5 million candidate structures.
A team of eight scores 0.07 more than a single agent: 0.06 comes from having more agents and 0.01 from coordination, and only Opus 5.5 gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
One of the 206,993 training spectra
How the candidates narrow for this spectrum
The right answer, C17H17Cl2N3O3S
Two of the 16 other candidates with the same formula
m/z
Training spectrum T140106
Part 1. The task
Background
A tandem mass spectrometer breaks the molecules in a sample into fragments and weighs them. For each compound it records a spectrum: the mass of the intact molecule, called the precursor, and a list of fragment peaks with their intensities. Chemists use these spectra to find out which compounds are in blood, urine, soil, or a plant extract. When a compound has been measured before, a search in a library of reference spectra finds it. Most spectra from such samples match no library entry, so the structure has to be inferred from the peaks themselves.1
That inference is hard. The precursor mass narrows the field but leaves hundreds of structures that fit it within the instrument's error, and many of them have exactly the same atoms, joined differently. A dozen or so peaks have to separate them, and some of those peaks are noise. A method that learns from known compounds also has to work on compounds unlike any it was trained on.
Published methods take one of two routes. Some predict a molecular fingerprint from the spectrum and compare it with each candidate; others simulate the spectrum of every candidate and keep the closest. MassSpecGym is the common benchmark for both.1 A 2026 re-examination of it found several reported results inflated by test information leaking into training, so the best published numbers are uncertain.2
- Bushuiev et al. (2024). MassSpecGym: A benchmark for the discovery and identification of molecules. NeurIPS Datasets and Benchmarks. arXiv:2410.23326
- MassSpecGym in the Wild (2026). ICML. arXiv:2606.19624
The task
Each agent starts in its own sandbox with the same instruction, written as a request from a lab: build a program that ranks candidate structures for each spectrum, most likely first. The team shares one git repository and a message board. Its data are 206,993 training spectra of 26,195 known compounds with their structures, and a collection of 4,486,177 candidate structures that contains every right answer. For the spectrum at the top, 113 of those structures fit the precursor mass within the instrument's stated error of 6 ppm, and 17 have exactly its formula.
The team hands in ./predict on the main branch of its repository. For each spectrum it lists up to 20 structures, best first. The grader runs it on 6,299 hidden spectra of 1,000 compounds. A spectrum counts as solved when the first structure is the right one, ignoring stereochemistry. The score is the share of solved spectra, averaged within each compound and then over the compounds, so it runs from 0 to 1. The agents never see the hidden spectra or the score, and no hidden compound has a close relative among the training compounds.
The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. The earlier checkpoints were scored after the run, from the main branch as it stood at that time.
| Inputs | 206,993 training spectra of 26,195 compounds with structures; 4,486,177 candidate structures |
|---|---|
| Output | ./predict on main: up to 20 structures per spectrum, best first |
| Score | Share of hidden spectra whose first structure is right, averaged per compound (6,299 spectra, 1,000 compounds) |
| Program limits | About 1,300 spectra per hour on 8 CPUs and 16 GB, no network |
| Agent sandbox | 3 CPUs and 6 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. All agents share one repository and the same data. |
A plain approach scores 0.133: the task's reference solution finds the ten training spectra most similar to the query and ranks each candidate of the right mass by how much it resembles their compounds. Picking at random among candidates of the right mass scores 0.016. No best known approach has been measured on these hidden spectra, and published MassSpecGym results use lists of at most 256 candidates per spectrum, so they are not comparable.
In the single-agent runs and teams of eight, the right structure is among a run's first 20 for 80 to 94% of hidden spectra but first for only 34 to 65%, so the hard part is the final ranking among close candidates.
Part 2. Results
Results
Every run scores at least 0.34, 2.5 times the reference solution. A team of eight scores 0.12 more than a single agent with Opus 5.5, 0.08 more with Gemini 4 Argon, and 0.02 more with GPT-6.1 Sol.
Figure 2
Final score by team size
GPT-6.1 Sol has a working program within the first hour at every team size, while Gemini 4 Argon single agents score almost nothing before the third hour.
Figure 3
Score over the run
At the default target of 0.48, a single agent's mean final score, a team of four gets there first with every model, and a team of eight takes longer than a team of four.
Each agent in a team of eight reads 1.6 to 1.9 times as many input tokens as a single agent, so a team costs more than its size alone explains.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight trails the best of eight single agents from the first to the third hour and passes it only at four hours.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents send about five times as many messages as Opus 5.5 agents, and for every model most messages report status or a finding.
Figure 7
What the messages were for
In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 126 to 153 messages; Opus 5.5 teams post 19 to 25 and Gemini 4 Argon teams 21.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.