Shared taskRNA structure
RNA Secondary Structure
Build a program that predicts which bases of an RNA pair with which, for RNA families it has never seen.
A team of eight scores 0.02 more than a single agent: 0.02 comes from having more agents and 0.00 from coordination, and only Opus 5.5 gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
One of the about 57,800 training RNAs: PhotoRC-II (Rfam RF01717), 95 nucleotides
- Nested pair, written ( )
- Crossing pair (pseudoknot), written [ ]
UAACCGUUGUGAACACCCCGUAAACCGAGACCUCUAGGGUGUCUAAAUCACGUCUCUAAUAUCCUGUCUUAGGGUGACAGGAAAUAGUAACUCCA .............(((((..((....[[[[[...)))))))..........]]]]].....(((((((.......))))))).............
The sequence is the program's input; the dot-bracket line is what it has to write.
Existing tools on the 1,496 hidden RNAs
train.parquet as the agents receive it, drawn as arcs between paired bases: 14 nested pairs and 5 crossing ones. The reference scores are the task's own, from tools that are not in the agents' image; the program is graded on RNAs from families held out of training.Training RNA Z35950.1/5070-5141
Part 1. The task
Background
An RNA molecule is a single chain of nucleotides, A, C, G and U, that folds back on itself. Stretches of the chain pair up, A with U, G with C and also G with U, and runs of stacked pairs form helices separated by loops. The list of which bases pair with which is the RNA's secondary structure. Much of what a structured RNA does depends on it, and it is the first step toward the molecule's shape in three dimensions.
A secondary structure can be written as a dot-bracket string: a dot for each unpaired base and a matching pair of brackets for each base pair. Most pairs nest inside one another. Pairs that cross the others form a pseudoknot and need a second kind of bracket. Measuring a structure is slow, so it is usually predicted. Single-sequence methods search for the fold with the lowest free energy, or for one that a model learned from known structures. Comparative methods align the sequence with related ones and look for pairs whose bases change together, which works better when relatives exist.
Rfam groups known RNAs into families, each with a seed alignment and a consensus structure, and groups related families into clans.1 The hard case is a family nobody has characterised. A method trained on known families has to carry over to it, and there may be few relatives to compare against.
- Rfam 15.1, the database the task's data are built from (CC0). ftp.ebi.ac.uk/pub/databases/Rfam/15.1
The task
Each agent starts in its own sandbox with the same instruction, written as a request: build a program that predicts an RNA's secondary structure from its sequence, to be run mostly on RNAs from families nobody has characterised. The team shares one git repository and a message board. Its data are about 57,800 RNAs with known structures, each tagged with its Rfam family and clan, and 284,731 Rfam sequences without structures that the program may search when it predicts. No RNA software is installed and the sandbox has no internet, so every team writes its own folding code, homology search and models.
The team hands in ./predict on the main branch of its repository. For each sequence of up to 1,000 nucleotides it writes a dot-bracket string, with () for ordinary pairs and [], {} or <> for pairs that cross them. The grader runs it on 1,496 hidden RNAs from 261 Rfam families held out of training, in 8 batches with at most one RNA per family. Each prediction scores the F1 of its base pairs against the true pairs, exact positions only. The score averages that within each family and then over the families, so it runs from 0 to 1. The agents never see the hidden RNAs or the score. No hidden family has a structure that Rfam made with a folding program, so running one on the sequence database cannot recover the labels.
The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. The earlier checkpoints were scored after the run, from the commit estimated to be on the main branch at that time.
| Inputs | About 57,800 RNAs with structures, Rfam family and clan; 284,731 sequences without structures, searchable at prediction time |
|---|---|
| Output | ./predict on main: one dot-bracket string per sequence, pseudoknots allowed |
| Score | Base-pair F1 per RNA, averaged within each family, then over 261 held-out families (1,496 RNAs) |
| Program limits | 8 batches on 4 CPUs and 8 GB, no network, a fresh copy of the data for each batch |
| Agent sandbox | 4 CPUs and 8 GB per agent, no internet, no RNA software |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. All agents share one repository and the same data. |
A greedy rule that keeps taking the longest helix that fits scores 0.33. Existing tools, none of them in the agents' image, set the bar: ViennaRNA's centroid fold scores 0.525, IPknot, the best single-sequence tool, 0.536, and an off-the-shelf comparative pipeline that searches the sequence database (BLASTN, MAFFT and RNAalifold) 0.56. Picking the best of 14 tools for each family with the help of the labels scores 0.635.
On the 55 hidden families with no relatives in the sequence database, 11 of the 12 model and team-size settings score below ViennaRNA's centroid fold on those families (0.551), so the teams add least where the instruction says the program will mostly run.
Part 2. Results
Results
Every run scores above IPknot's 0.536, and 49 of the 51 runs above the comparative pipeline's 0.56. A team of eight scores 0.04 more than a single agent with Opus 5.5, and 0.01 more with GPT-6.1 Sol and with Gemini 4 Argon.
Figure 2
Final score by team size
Teams of eight are ahead of single agents at 30 minutes with every model, when Gemini 4 Argon single agents have nothing on main yet. After two hours every setting is within 0.04 of its final score.
Figure 3
Score over the run
At the default target of 0.61, a single agent's mean final score, teams of four and eight get there in under two hours with every model, and Opus 5.5 single agents never do.
A team of eight costs 10.1 to 12.0 times as much as a single agent, more than its size alone explains, for a gain of 0.01 to 0.04.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight leads the best of eight single agents at 30 minutes, trails it at one hour, and from two hours on the two are within 0.01 of each other.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents send three to four times as many messages per hour as Opus 5.5 agents, most of them to named teammates. For every model most messages report status or a finding.
Figure 7
What the messages were for
In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 108 to 130 messages; Opus 5.5 teams post 31 to 35 and Gemini 4 Argon teams 14 to 21.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.