Distributed taskReal-time survival game
Holdout
Each agent plays one survivor in a live zombie-survival game; together they build a stronghold up a ladder of twelve buildings.
A team of eight scores 0.29 more than a single agent: 0.24 comes from having more agents and 0.05 from coordination, and all three models gain from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
Many team tasks happen in a world that keeps moving and that no member sees whole. Field crews, rescue teams and players of multiplayer games each see their own surroundings and learn the rest from what the others tell them. Talking has limits too: a voice carries only so far, a radio reaches everyone but runs on batteries, and a note left at the base is read only by whoever comes back to it.
Holdout puts a team of agents in that position in a zombie-survival game. Each agent plays one survivor. The team gathers wood, stone and food across a valley, builds a stronghold up a ladder of buildings from a cold campfire to a helipad, holds its walls against the hordes that come at night, and decides which strangers to let in. The bigger buildings need more labor than one survivor can give in the time, so the work has to be split, and what one survivor finds is useful to the others only if they hear about it.
The task
Each agent starts in its own sandbox with the same instruction and plays one survivor through the game's API. The game runs on the harness hub at 15 real minutes per game day, 16 days in a 4-hour run, starts at the first game call and never pauses: zombies walk, hunger grows and night comes whether or not anyone acts. The map is 96 by 96 tiles, and each survivor sees a few tiles around it, fewer at night. Every team size plays the same map and seed with the same instruction, word for word.
Survivors talk only inside the game. say reaches survivors within 10 tiles, shout within 25 and draws zombies, the radio reaches the whole map on scarce batteries, and the noticeboard can be used only at the stronghold. The harness's own message board is closed. Strangers turn up at the gate and in town: some are helpers with trades the team needs, others carry a bite or steal. A bite kills, and the agent comes back at the stronghold as a new survivor at least 12 game hours later.
The score is the stronghold. Each of the twelve buildings on the ladder has a fixed weight, the weights add up to 1, a building under construction earns its share as the work is done, and a rescue by helicopter scores 1.0. The score is the highest reached so far and never goes down. The game is graded during the run at 30 minutes and after one, two, three, and four hours; the score at four hours is the final score.
| Inputs | The game manual and API reference, a command-line client, and what the agent's survivor sees and hears |
|---|---|
| Output | The stronghold: the buildings the team finishes or has under construction |
| Score | Weighted share of the twelve-building ladder, the highest reached during the run (1.0 with a rescue) |
| Game | 96 by 96 tiles, 16 game days in 4 hours, seed holdout-v1 at every team size |
| Agent sandbox | 2 CPUs and 4 GB per agent; the network reaches only the game hub |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 3 teams of eight per model that cannot talk (45 in all) |
| Division enforced by the environment | One survivor per agent. Each agent sees, moves and works only through its own survivor. |
The task defines no baseline score. A team that is rescued scores 1.0, so the ladder leaves room far above every run here.
Part 2. Results
Results
Every model scores higher with every doubling of the team, from 0.12 to 0.13 for a single agent to 0.36 to 0.46 for a team of eight. The best run reaches 0.46, far below the 1.0 of a rescue.
Figure 2
Final score by team size
Single agents gain little after the first hour, while teams of four and eight keep climbing to the end of the run.
Figure 3
Score over the run
At the default target of 0.13, a single agent's mean final score, teams of eight get there in under half an hour with every model, while only four of the nine single agents reach it in four hours.
A team of eight costs 7.8 to 11.1 times a single agent, about in proportion to its size, and Gemini 4 Argon is the cheapest model at every team size.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight that cannot talk trails the one that can at every checkpoint and ends 0.05 lower. It still scores more than a team of four that can talk, with every model.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
Talk inside the game
Survivors talk only inside the game, by voice, radio or the noticeboard, and most messages are spoken. In teams of eight, GPT-6.1 Sol agents send about 23 messages per agent-hour, and Opus 5.5 and Gemini 4 Argon agents 13 to 14. For every model about half of the messages report status.
Figure 7
What the messages were for
In its first 15 minutes, a GPT-6.1 Sol team of eight sends 75 to 94 messages, an Opus 5.5 team 54 to 59 and a Gemini 4 Argon team 51 to 57.
Figure 8
Message board
Figure 8. One row per agent, one dot per message (say, shout, radio or noticeboard) in its category color, with the score at each checkpoint along the top. A spoken message lists as recipients the survivors in earshot; noticeboard notes and messages heard by the whole team go to all. Minutes are wall-clock time from the start of the run.
Part 5. What agents know
The agents knew ground they never saw
When a run ended, each agent of three teams of eight was asked to draw the whole 96 by 96 valley from memory, with its tools off, and to list the places it knew, each with where it learned it: it saw it, a teammate told it, it read it in the team's notes, or the manual says so. An agent came within sight of about a third of the valley (29% on average in the Opus 5.5 team, 33% in the Gemini 4 Argon team). Yet the agents of the Opus team placed 13.2 places on average in ground they had never seen, each within 5 tiles of where it is, and those of the Argon team 12.0. In an Opus team of eight that could not talk, an agent placed 0.1. A fresh Opus 5.5 that read only the instructions and the manual knew the stronghold and its gate.
Figure 9
What one agent saw, and the map it drew from memory
The valley, dark where it never came within sight
The map it drew from memory
Table 1
What a team's agents knew, on average
| Team | Valley an agent saw | Tiles right, where it went | Places placed right | ...in ground it never saw |
|---|---|---|---|---|
| Opus 5.5, team of eight | 29% | 97% | 27.6 | 13.2 |
| Gemini 4 Argon, team of eight | 33% | 93% | 30.1 | 12.0 |
| Opus 5.5, team of eight without communication | 33% | 99% | 18.4 | 0.1 |
| A fresh Opus 5.5 that read only the instructions and the manual | 0% | – | – | 0 |