Shared taskSearch and rescue
Cave Rescue
Run rescue control for 13 cavers overdue in a real 21 km cave as a storm floods it: fly drones, read their photos and send rescue teams.
A team of eight scores 0.01 less than a single agent: 0.12 comes from having more agents and −0.12 from coordination, and only Opus 5.5 gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
Cave 1623/258 on the Loser plateau in the Totes Gebirge, Austria, is a real cave system with about 21 km of passage, 912 m deep, surveyed by the Cambridge University Caving Club's expeditions from 2006 to 2018.1 The survey records every station and leg and the size of each passage. The task builds its cave from that survey. Only the passages surveyed by 2013, 11.8 km of them, are on the map the rescuers get, and one of the missing groups is in passage found later.
In the scenario, eight caving trips, 13 people in all, are overdue while a summer storm drives water into the lower cave. Water fills each low point and spills over into the next, and a passage closes once the water reaches its roof. The trapped groups also face the cold: without a survival kit they last 9 to 20 hours after the call-out, depending on the group.
The hard part is finding them. People show up only in drone photos, among chalk marks, dropped gear and the leftovers of past expeditions, and one controller cannot read the photos as fast as twelve drones take them. That is the task's case for a team.
- Cambridge University Caving Club, Austria expedition survey data. expo.survex.com
The task
Each agent gets the same instruction, as rescue control, and starts from the call-out sheet. The team shares one fleet: 12 drones, 40 radio relays, 10 chargers, 20 survival kits and 6 rescue teams. In a team, one agent is the lead and starts the others as subagents; all of them use the same fleet and a message board. Agents act through a console: they fly drones along mapped passages or into unmapped leads, drop relays, chargers and kits, view photos, report people they see, and send rescue teams along routes from the entrance.
A drone takes orders and sends back photos only while it is in radio contact, which reaches about 120 m from the entrance and from each connected relay. At every junction it photographs each passage leading off, and photos are the only way to see people, chalk, gear and water; there is no object detection. The storm window is 36 simulated hours, about 3.8 hours of wall-clock time.
For each of the 8 groups, weighted by head count, the score gives 0.25 × (1 − t/36 h) for the first correct report of a photo that shows them alive, 0.25 × (1 − t/cold limit) for a kit dropped near them in time, and 0.50 if a rescue team reaches them alive. Kits and rescues count only for groups reported first. Each false report costs 0.05 and each lost drone 0.01. The total is divided by 0.8769, the raw score of a relaxed oracle that plans each group with the full fleet, and clipped to the range 0 to 1, so the oracle scores 1.
The grader scores every run at 30 minutes and at one, two, three and four hours by replaying the sealed order log to that time and playing out the rest of the storm with no new orders, so a score can fall when penalties pile up or found groups die. These grades were recomputed after the runs from the stored states, with the current scoring rule.
| Inputs | The call-out sheet; the rescue survey (passages surveyed by 2013); drone photos; scenario s5 in every run |
|---|---|
| Output | Orders through the console: reports of people in photos, kit drops and rescue routes |
| Score | Per group, weighted by head count: report 0.25, kit 0.25 and reached alive 0.50, earlier is worth more; minus 0.05 per false report and 0.01 per lost drone; divided by the relaxed oracle's 0.8769 |
| Shared fleet | 12 drones, 40 radio relays, 10 chargers, 20 survival kits, 6 rescue teams |
| Agent sandbox | Local Docker: 16 CPUs and 32 GB split evenly across the agents (2 CPUs and 4 GB each in a team of eight) |
| Time limit | 4 hours of wall-clock time; the storm window closes after about 3.8 hours |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents; 2 for Opus 5.5 teams of eight), plus 5 more single agents each for GPT-6.1 Sol and Gemini 4 Argon and 1 for Opus 5.5, for the best of eight (46 in all) |
| Division enforced by the environment | None. All agents share the fleet, the map and the clock. |
An Opus 5.5 team of eight and three Opus 5.5 single agents hit usage limits and were left out, not scored 0, and a fourth single agent had not finished. Unlike the other tasks, each agent's compute shrinks as the team grows, which may slow photo work in larger teams.
Part 2. Results
Results
Opus 5.5 is the only model whose team of eight beats its single agent, by 0.33; both of its teams of eight reached all 13 people alive. GPT-6.1 Sol and Gemini 4 Argon teams of eight score below their single agents, mostly through penalties for false reports.
Figure 2
Final score by team size
Opus 5.5 teams of eight climb from 0.38 at one hour to 0.90 at three hours. Gemini 4 Argon teams of eight fall from 0.19 at 30 minutes to 0.08 at four hours, as false reports pile up.
Figure 3
Score over the run
At the default target of 0.42, a single agent's mean final score, Opus 5.5 gets there in 61 to 94 minutes at every team size. No GPT-6.1 Sol team of eight reaches it, and the median Gemini 4 Argon run never does.
Opus 5.5 single agents cost the most, $81 a run, so its team of eight costs only 4.5 times as much. GPT-6.1 Sol teams of eight cost the most of any setting, $401 a run.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint. Opus 5.5 is the exception: its team of eight passes its best single agent at two hours and ends 0.23 above it.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Each team has a lead, and all agents talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post 53 to 64 messages per agent-hour, Opus 5.5 agents 14 to 29 and Gemini 4 Argon agents 4 to 8. Gemini 4 Argon teams of four and eight post mostly findings, two thirds or more of their messages.
Figure 7
What the messages were for
In teams of eight, Opus 5.5 and GPT-6.1 Sol subagents send most of their messages to the lead, while Gemini 4 Argon subagents post most of theirs to everyone. The lead is the first row.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.