← MAC Bench

Shared taskSearch and rescue

Cave Rescue

Run rescue control for 13 cavers overdue in a real 21 km cave as a storm floods it: fly drones, read their photos and send rescue teams.

Bespoke Labs3 models, 4 team sizes, 46 runs

A team of eight scores 0.01 less than a single agent: 0.12 comes from having more agents and −0.12 from coordination, and only Opus 5.5 gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of two or three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score; for Opus 5.5 it is the best of four, because four of its single-agent runs are missing. The number after each bar is the score of the team of eight; point at a row for its parts.
A 3D replay of the cave, rebuilt from the order log of a team of two Opus 5.5 agents, in a scenario with three missing groups. Drag to turn the cave, and use the slider to move through the storm.

Background

Cave 1623/258 on the Loser plateau in the Totes Gebirge, Austria, is a real cave system with about 21 km of passage, 912 m deep, surveyed by the Cambridge University Caving Club's expeditions from 2006 to 2018.1 The survey records every station and leg and the size of each passage. The task builds its cave from that survey. Only the passages surveyed by 2013, 11.8 km of them, are on the map the rescuers get, and one of the missing groups is in passage found later.

In the scenario, eight caving trips, 13 people in all, are overdue while a summer storm drives water into the lower cave. Water fills each low point and spills over into the next, and a passage closes once the water reaches its roof. The trapped groups also face the cold: without a survival kit they last 9 to 20 hours after the call-out, depending on the group.

The hard part is finding them. People show up only in drone photos, among chalk marks, dropped gear and the leftovers of past expeditions, and one controller cannot read the photos as fast as twelve drones take them. That is the task's case for a team.

  1. Cambridge University Caving Club, Austria expedition survey data. expo.survex.com

The task

Each agent gets the same instruction, as rescue control, and starts from the call-out sheet. The team shares one fleet: 12 drones, 40 radio relays, 10 chargers, 20 survival kits and 6 rescue teams. In a team, one agent is the lead and starts the others as subagents; all of them use the same fleet and a message board. Agents act through a console: they fly drones along mapped passages or into unmapped leads, drop relays, chargers and kits, view photos, report people they see, and send rescue teams along routes from the entrance.

A drone takes orders and sends back photos only while it is in radio contact, which reaches about 120 m from the entrance and from each connected relay. At every junction it photographs each passage leading off, and photos are the only way to see people, chalk, gear and water; there is no object detection. The storm window is 36 simulated hours, about 3.8 hours of wall-clock time.

For each of the 8 groups, weighted by head count, the score gives 0.25 × (1 − t/36 h) for the first correct report of a photo that shows them alive, 0.25 × (1 − t/cold limit) for a kit dropped near them in time, and 0.50 if a rescue team reaches them alive. Kits and rescues count only for groups reported first. Each false report costs 0.05 and each lost drone 0.01. The total is divided by 0.8769, the raw score of a relaxed oracle that plans each group with the full fleet, and clipped to the range 0 to 1, so the oracle scores 1.

The grader scores every run at 30 minutes and at one, two, three and four hours by replaying the sealed order log to that time and playing out the rest of the storm with no new orders, so a score can fall when penalties pile up or found groups die. These grades were recomputed after the runs from the stored states, with the current scoring rule.

InputsThe call-out sheet; the rescue survey (passages surveyed by 2013); drone photos; scenario s5 in every run
OutputOrders through the console: reports of people in photos, kit drops and rescue routes
ScorePer group, weighted by head count: report 0.25, kit 0.25 and reached alive 0.50, earlier is worth more; minus 0.05 per false report and 0.01 per lost drone; divided by the relaxed oracle's 0.8769
Shared fleet12 drones, 40 radio relays, 10 chargers, 20 survival kits, 6 rescue teams
Agent sandboxLocal Docker: 16 CPUs and 32 GB split evenly across the agents (2 CPUs and 4 GB each in a team of eight)
Time limit4 hours of wall-clock time; the storm window closes after about 3.8 hours
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents; 2 for Opus 5.5 teams of eight), plus 5 more single agents each for GPT-6.1 Sol and Gemini 4 Argon and 1 for Opus 5.5, for the best of eight (46 in all)
Division enforced by the environmentNone. All agents share the fleet, the map and the clock.

An Opus 5.5 team of eight and three Opus 5.5 single agents hit usage limits and were left out, not scored 0, and a fourth single agent had not finished. Unlike the other tasks, each agent's compute shrinks as the team grows, which may slow photo work in larger teams.

Results

Opus 5.5 is the only model whose team of eight beats its single agent, by 0.33; both of its teams of eight reached all 13 people alive. GPT-6.1 Sol and Gemini 4 Argon teams of eight score below their single agents, mostly through penalties for false reports.

Figure 2

Final score by team size

Figure 2. Dots are runs, two or three per setting; the bar is the mean.

Opus 5.5 teams of eight climb from 0.38 at one hour to 0.90 at three hours. Gemini 4 Argon teams of eight fall from 0.19 at 30 minutes to 0.08 at four hours, as false reports pile up.

Figure 3

Score over the run

Figure 3. Mean of two or three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Each checkpoint plays out the rest of the storm with no new orders; a team that finished early keeps its final score.

At the default target of 0.42, a single agent's mean final score, Opus 5.5 gets there in 61 to 94 minutes at every team size. No GPT-6.1 Sol team of eight reaches it, and the median Gemini 4 Argon run never does.

Figure 4

Time to reach a score

0.42
00.250.42 single agent0.751

    Figure 4. Large marks are the median over two or three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    Opus 5.5 single agents cost the most, $81 a run, so its team of eight costs only 4.5 times as much. GPT-6.1 Sol teams of eight cost the most of any setting, $401 a run.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, two or three per setting, on a log scale; dotted lines join the means. A team of eight costs 4.5× a single agent with Opus 5.5, 7.2× with Sol, and 8.6× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint. Opus 5.5 is the exception: its team of eight passes its best single agent at two hours and ends 0.23 above it.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of two or three runs; without coordination is the best of the eight single agents at each checkpoint (the best of four for Opus 5.5).

    What the messages were for

    Each team has a lead, and all agents talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post 53 to 64 messages per agent-hour, Opus 5.5 agents 14 to 29 and Gemini 4 Argon agents 4 to 8. Gemini 4 Argon teams of four and eight post mostly findings, two thirds or more of their messages.

    Figure 7

    What the messages were for

      Figure 7. Runs pooled per row: three, or two for the Opus 5.5 team of eight; every message is labeled. Claude Sonnet 5.5 labeled them from the start of each message as the board shows it (up to 300 characters), with the same category definitions as the other MAC Bench tasks; a blind check by Claude Opus 5.5 agreed on 77% of 410.

      In teams of eight, Opus 5.5 and GPT-6.1 Sol subagents send most of their messages to the lead, while Gemini 4 Argon subagents post most of theirs to everyone. The lead is the first row.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.