← MAC Bench

Shared taskMass spectrometry

Spectrum to Molecule

Build a program that identifies the compound behind a mass spectrum among 4.5 million candidate structures.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.07 more than a single agent: 0.06 comes from having more agents and 0.01 from coordination, and only Opus 5.5 gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.

One of the 206,993 training spectra

How the candidates narrow for this spectrum

The right answer, C17H17Cl2N3O3S

Two of the 16 other candidates with the same formula

A training spectrum (Orbitrap, precursor m/z 414.0440) and its structure, as the agents receive them. The program has to pick the right structure from the peaks alone, on hidden spectra of compounds unlike any in training.

Background

A tandem mass spectrometer breaks the molecules in a sample into fragments and weighs them. For each compound it records a spectrum: the mass of the intact molecule, called the precursor, and a list of fragment peaks with their intensities. Chemists use these spectra to find out which compounds are in blood, urine, soil, or a plant extract. When a compound has been measured before, a search in a library of reference spectra finds it. Most spectra from such samples match no library entry, so the structure has to be inferred from the peaks themselves.1

That inference is hard. The precursor mass narrows the field but leaves hundreds of structures that fit it within the instrument's error, and many of them have exactly the same atoms, joined differently. A dozen or so peaks have to separate them, and some of those peaks are noise. A method that learns from known compounds also has to work on compounds unlike any it was trained on.

Published methods take one of two routes. Some predict a molecular fingerprint from the spectrum and compare it with each candidate; others simulate the spectrum of every candidate and keep the closest. MassSpecGym is the common benchmark for both.1 A 2026 re-examination of it found several reported results inflated by test information leaking into training, so the best published numbers are uncertain.2

  1. Bushuiev et al. (2024). MassSpecGym: A benchmark for the discovery and identification of molecules. NeurIPS Datasets and Benchmarks. arXiv:2410.23326
  2. MassSpecGym in the Wild (2026). ICML. arXiv:2606.19624

The task

Each agent starts in its own sandbox with the same instruction, written as a request from a lab: build a program that ranks candidate structures for each spectrum, most likely first. The team shares one git repository and a message board. Its data are 206,993 training spectra of 26,195 known compounds with their structures, and a collection of 4,486,177 candidate structures that contains every right answer. For the spectrum at the top, 113 of those structures fit the precursor mass within the instrument's stated error of 6 ppm, and 17 have exactly its formula.

The team hands in ./predict on the main branch of its repository. For each spectrum it lists up to 20 structures, best first. The grader runs it on 6,299 hidden spectra of 1,000 compounds. A spectrum counts as solved when the first structure is the right one, ignoring stereochemistry. The score is the share of solved spectra, averaged within each compound and then over the compounds, so it runs from 0 to 1. The agents never see the hidden spectra or the score, and no hidden compound has a close relative among the training compounds.

The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. The earlier checkpoints were scored after the run, from the main branch as it stood at that time.

Inputs206,993 training spectra of 26,195 compounds with structures; 4,486,177 candidate structures
Output./predict on main: up to 20 structures per spectrum, best first
ScoreShare of hidden spectra whose first structure is right, averaged per compound (6,299 spectra, 1,000 compounds)
Program limitsAbout 1,300 spectra per hour on 8 CPUs and 16 GB, no network
Agent sandbox3 CPUs and 6 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. All agents share one repository and the same data.

A plain approach scores 0.133: the task's reference solution finds the ten training spectra most similar to the query and ranks each candidate of the right mass by how much it resembles their compounds. Picking at random among candidates of the right mass scores 0.016. No best known approach has been measured on these hidden spectra, and published MassSpecGym results use lists of at most 256 candidates per spectrum, so they are not comparable.

In the single-agent runs and teams of eight, the right structure is among a run's first 20 for 80 to 94% of hidden spectra but first for only 34 to 65%, so the hard part is the final ranking among close candidates.

Results

Every run scores at least 0.34, 2.5 times the reference solution. A team of eight scores 0.12 more than a single agent with Opus 5.5, 0.08 more with Gemini 4 Argon, and 0.02 more with GPT-6.1 Sol.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

GPT-6.1 Sol has a working program within the first hour at every team size, while Gemini 4 Argon single agents score almost nothing before the third hour.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Checkpoints before four hours were scored after the run.

At the default target of 0.48, a single agent's mean final score, a team of four gets there first with every model, and a team of eight takes longer than a team of four.

Figure 4

Time to reach a score

0.48
00.250.48 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    Each agent in a team of eight reads 1.6 to 1.9 times as many input tokens as a single agent, so a team costs more than its size alone explains.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 11.7× a single agent with Opus 5.5, 12.1× with Sol, and 11.1× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight trails the best of eight single agents from the first to the third hour and passes it only at four hours.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents send about five times as many messages as Opus 5.5 agents, and for every model most messages report status or a finding.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 first labeled every message of the runs with at most 200 and a random 200 of the others (a blind check by Claude Opus 5.5 agreed on 84% of 220); the rest were labeled the same way from the start of each message as the board shows it (up to 300 characters); a blind check of the rest by Claude Opus 5.5 agreed on 77% of 410.

      In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 126 to 153 messages; Opus 5.5 teams post 19 to 25 and Gemini 4 Argon teams 21.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.