← MAC Bench

Distributed taskReal-time survival game

Holdout

Each agent plays one survivor in a live zombie-survival game; together they build a stronghold up a ladder of twelve buildings.

Bespoke Labs3 models, 4 team sizes, 45 runs

A team of eight scores 0.29 more than a single agent: 0.24 comes from having more agents and 0.05 from coordination, and all three models gain from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this distributed task, eight agents without coordination is a team of eight whose survivors cannot talk: the game refuses say, shout, radio and the noticeboard, and git pushes to the team repository are closed (mean of three runs per model). The number after each bar is the score of the team of eight; point at a row for its parts.
The task's own replay viewer, with one run of Opus 5.5 and one of Gemini 4 Argon at each team size, the first of their three rollouts. Pick a model and a team; the messages the survivors send run down the right as the game plays. Drag to pan, scroll to zoom, and click a survivor to follow them.

Background

Many team tasks happen in a world that keeps moving and that no member sees whole. Field crews, rescue teams and players of multiplayer games each see their own surroundings and learn the rest from what the others tell them. Talking has limits too: a voice carries only so far, a radio reaches everyone but runs on batteries, and a note left at the base is read only by whoever comes back to it.

Holdout puts a team of agents in that position in a zombie-survival game. Each agent plays one survivor. The team gathers wood, stone and food across a valley, builds a stronghold up a ladder of buildings from a cold campfire to a helipad, holds its walls against the hordes that come at night, and decides which strangers to let in. The bigger buildings need more labor than one survivor can give in the time, so the work has to be split, and what one survivor finds is useful to the others only if they hear about it.

The task

Each agent starts in its own sandbox with the same instruction and plays one survivor through the game's API. The game runs on the harness hub at 15 real minutes per game day, 16 days in a 4-hour run, starts at the first game call and never pauses: zombies walk, hunger grows and night comes whether or not anyone acts. The map is 96 by 96 tiles, and each survivor sees a few tiles around it, fewer at night. Every team size plays the same map and seed with the same instruction, word for word.

Survivors talk only inside the game. say reaches survivors within 10 tiles, shout within 25 and draws zombies, the radio reaches the whole map on scarce batteries, and the noticeboard can be used only at the stronghold. The harness's own message board is closed. Strangers turn up at the gate and in town: some are helpers with trades the team needs, others carry a bite or steal. A bite kills, and the agent comes back at the stronghold as a new survivor at least 12 game hours later.

The score is the stronghold. Each of the twelve buildings on the ladder has a fixed weight, the weights add up to 1, a building under construction earns its share as the work is done, and a rescue by helicopter scores 1.0. The score is the highest reached so far and never goes down. The game is graded during the run at 30 minutes and after one, two, three, and four hours; the score at four hours is the final score.

InputsThe game manual and API reference, a command-line client, and what the agent's survivor sees and hears
OutputThe stronghold: the buildings the team finishes or has under construction
ScoreWeighted share of the twelve-building ladder, the highest reached during the run (1.0 with a rescue)
Game96 by 96 tiles, 16 game days in 4 hours, seed holdout-v1 at every team size
Agent sandbox2 CPUs and 4 GB per agent; the network reaches only the game hub
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 3 teams of eight per model that cannot talk (45 in all)
Division enforced by the environmentOne survivor per agent. Each agent sees, moves and works only through its own survivor.

The task defines no baseline score. A team that is rescued scores 1.0, so the ladder leaves room far above every run here.

Results

Every model scores higher with every doubling of the team, from 0.12 to 0.13 for a single agent to 0.36 to 0.46 for a team of eight. The best run reaches 0.46, far below the 1.0 of a rescue.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Single agents gain little after the first hour, while teams of four and eight keep climbing to the end of the run.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. The game was graded during the run at each checkpoint.

At the default target of 0.13, a single agent's mean final score, teams of eight get there in under half an hour with every model, while only four of the nine single agents reach it in four hours.

Figure 4

Time to reach a score

0.13
00.250.13 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 7.8 to 11.1 times a single agent, about in proportion to its size, and Gemini 4 Argon is the cheapest model at every team size.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 11.1× a single agent with Opus 5.5, 8.1× with Sol, and 7.8× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight that cannot talk trails the one that can at every checkpoint and ends 0.05 lower. It still scores more than a team of four that can talk, with every model.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the mean of the teams of eight whose in-game channels and git pushes were closed.

    Talk inside the game

    Survivors talk only inside the game, by voice, radio or the noticeboard, and most messages are spoken. In teams of eight, GPT-6.1 Sol agents send about 23 messages per agent-hour, and Opus 5.5 and Gemini 4 Argon agents 13 to 14. For every model about half of the messages report status.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 labeled them from the start of each message as the board shows it (up to 300 characters), with the same category definitions as the other MAC Bench tasks; a blind check by Claude Opus 5.5 agreed on 77% of 410.

      In its first 15 minutes, a GPT-6.1 Sol team of eight sends 75 to 94 messages, an Opus 5.5 team 54 to 59 and a Gemini 4 Argon team 51 to 57.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message (say, shout, radio or noticeboard) in its category color, with the score at each checkpoint along the top. A spoken message lists as recipients the survivors in earshot; noticeboard notes and messages heard by the whole team go to all. Minutes are wall-clock time from the start of the run.

          The agents knew ground they never saw

          When a run ended, each agent of three teams of eight was asked to draw the whole 96 by 96 valley from memory, with its tools off, and to list the places it knew, each with where it learned it: it saw it, a teammate told it, it read it in the team's notes, or the manual says so. An agent came within sight of about a third of the valley (29% on average in the Opus 5.5 team, 33% in the Gemini 4 Argon team). Yet the agents of the Opus team placed 13.2 places on average in ground they had never seen, each within 5 tiles of where it is, and those of the Argon team 12.0. In an Opus team of eight that could not talk, an agent placed 0.1. A fresh Opus 5.5 that read only the instructions and the manual knew the stronghold and its gate.

          Figure 9

          What one agent saw, and the map it drew from memory

          The valley, dark where it never came within sight

          The map it drew from memory

            Figure 9. The first map is the real valley, dark wherever the chosen agent never came within sight (7 tiles by day, 4 at night), with its path in yellow. The second is the map it drew at the end of the run, from its own session with no tools; dark is ground it marked unknown, and each white ring is a place it named. A tile it filled in where it never went came from a teammate, the team's notes, the manual or a guess.

            Table 1

            What a team's agents knew, on average

            TeamValley an agent sawTiles right, where it wentPlaces placed right...in ground it never saw
            Opus 5.5, team of eight29%97%27.613.2
            Gemini 4 Argon, team of eight33%93%30.112.0
            Opus 5.5, team of eight without communication33%99%18.40.1
            A fresh Opus 5.5 that read only the instructions and the manual0%––0
            Table 1. Averages over the eight agents of a team. "Tiles right" is the share of the tiles an agent filled in, on ground it saw, that match the real map exactly; most of the valley is grass, so this share is high for every team. A place counts as placed right when it is within 5 tiles of the real one. The fresh Opus 5.5 named only the stronghold and its gate, from the manual; the stronghold is not one of the landmarks counted here. The teams with communication are the runs shown in the replay above; the team without communication is the first of Opus 5.5's three teams of eight without coordination.