← MAC Bench

Shared taskRNA structure

RNA Secondary Structure

Build a program that predicts which bases of an RNA pair with which, for RNA families it has never seen.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.02 more than a single agent: 0.02 comes from having more agents and 0.00 from coordination, and only Opus 5.5 gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.

One of the about 57,800 training RNAs: PhotoRC-II (Rfam RF01717), 95 nucleotides

  • Nested pair, written ( )
  • Crossing pair (pseudoknot), written [ ]
UAACCGUUGUGAACACCCCGUAAACCGAGACCUCUAGGGUGUCUAAAUCACGUCUCUAAUAUCCUGUCUUAGGGUGACAGGAAAUAGUAACUCCA
.............(((((..((....[[[[[...)))))))..........]]]]].....(((((((.......))))))).............

The sequence is the program's input; the dot-bracket line is what it has to write.

Existing tools on the 1,496 hidden RNAs

A row of train.parquet as the agents receive it, drawn as arcs between paired bases: 14 nested pairs and 5 crossing ones. The reference scores are the task's own, from tools that are not in the agents' image; the program is graded on RNAs from families held out of training.

Background

An RNA molecule is a single chain of nucleotides, A, C, G and U, that folds back on itself. Stretches of the chain pair up, A with U, G with C and also G with U, and runs of stacked pairs form helices separated by loops. The list of which bases pair with which is the RNA's secondary structure. Much of what a structured RNA does depends on it, and it is the first step toward the molecule's shape in three dimensions.

A secondary structure can be written as a dot-bracket string: a dot for each unpaired base and a matching pair of brackets for each base pair. Most pairs nest inside one another. Pairs that cross the others form a pseudoknot and need a second kind of bracket. Measuring a structure is slow, so it is usually predicted. Single-sequence methods search for the fold with the lowest free energy, or for one that a model learned from known structures. Comparative methods align the sequence with related ones and look for pairs whose bases change together, which works better when relatives exist.

Rfam groups known RNAs into families, each with a seed alignment and a consensus structure, and groups related families into clans.1 The hard case is a family nobody has characterised. A method trained on known families has to carry over to it, and there may be few relatives to compare against.

  1. Rfam 15.1, the database the task's data are built from (CC0). ftp.ebi.ac.uk/pub/databases/Rfam/15.1

The task

Each agent starts in its own sandbox with the same instruction, written as a request: build a program that predicts an RNA's secondary structure from its sequence, to be run mostly on RNAs from families nobody has characterised. The team shares one git repository and a message board. Its data are about 57,800 RNAs with known structures, each tagged with its Rfam family and clan, and 284,731 Rfam sequences without structures that the program may search when it predicts. No RNA software is installed and the sandbox has no internet, so every team writes its own folding code, homology search and models.

The team hands in ./predict on the main branch of its repository. For each sequence of up to 1,000 nucleotides it writes a dot-bracket string, with () for ordinary pairs and [], {} or <> for pairs that cross them. The grader runs it on 1,496 hidden RNAs from 261 Rfam families held out of training, in 8 batches with at most one RNA per family. Each prediction scores the F1 of its base pairs against the true pairs, exact positions only. The score averages that within each family and then over the families, so it runs from 0 to 1. The agents never see the hidden RNAs or the score. No hidden family has a structure that Rfam made with a folding program, so running one on the sequence database cannot recover the labels.

The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. The earlier checkpoints were scored after the run, from the commit estimated to be on the main branch at that time.

InputsAbout 57,800 RNAs with structures, Rfam family and clan; 284,731 sequences without structures, searchable at prediction time
Output./predict on main: one dot-bracket string per sequence, pseudoknots allowed
ScoreBase-pair F1 per RNA, averaged within each family, then over 261 held-out families (1,496 RNAs)
Program limits8 batches on 4 CPUs and 8 GB, no network, a fresh copy of the data for each batch
Agent sandbox4 CPUs and 8 GB per agent, no internet, no RNA software
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. All agents share one repository and the same data.

A greedy rule that keeps taking the longest helix that fits scores 0.33. Existing tools, none of them in the agents' image, set the bar: ViennaRNA's centroid fold scores 0.525, IPknot, the best single-sequence tool, 0.536, and an off-the-shelf comparative pipeline that searches the sequence database (BLASTN, MAFFT and RNAalifold) 0.56. Picking the best of 14 tools for each family with the help of the labels scores 0.635.

On the 55 hidden families with no relatives in the sequence database, 11 of the 12 model and team-size settings score below ViennaRNA's centroid fold on those families (0.551), so the teams add least where the instruction says the program will mostly run.

Results

Every run scores above IPknot's 0.536, and 49 of the 51 runs above the comparative pipeline's 0.56. A team of eight scores 0.04 more than a single agent with Opus 5.5, and 0.01 more with GPT-6.1 Sol and with Gemini 4 Argon.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Teams of eight are ahead of single agents at 30 minutes with every model, when Gemini 4 Argon single agents have nothing on main yet. After two hours every setting is within 0.04 of its final score.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Checkpoints before four hours were scored after the run.

At the default target of 0.61, a single agent's mean final score, teams of four and eight get there in under two hours with every model, and Opus 5.5 single agents never do.

Figure 4

Time to reach a score

0.61
00.250.61 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 10.1 to 12.0 times as much as a single agent, more than its size alone explains, for a gain of 0.01 to 0.04.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 10.3× a single agent with Opus 5.5, 12.0× with Sol, and 10.1× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight leads the best of eight single agents at 30 minutes, trails it at one hour, and from two hours on the two are within 0.01 of each other.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents send three to four times as many messages per hour as Opus 5.5 agents, most of them to named teammates. For every model most messages report status or a finding.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 labeled them from the start of each message as the board shows it (up to 300 characters), with the same category definitions as the other MAC Bench tasks; a blind check by Claude Opus 5.5 agreed on 77% of 410.

      In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 108 to 130 messages; Opus 5.5 teams post 31 to 35 and Gemini 4 Argon teams 14 to 21.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.