We release Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models.
Bespoke LabsNovember 18, 2026
Leaderboard
single agent
gain from more agents
gain from coordination
loss from coordination
team of eight agents
Opus 5.50.534GPT-6.1 Sol0.406Gemini 4 Argon0.319
00.20.40.60.81
mean final score
Introduction
In September 2026, OpenAI deployed approximately 10,000 agents to solve the Navier-Stokes existence and smoothness problem, arriving at a solution in just 88 hours. Scaling the number of agents is emerging as a new axis for scaling test-time compute (Park et al., 2026). But more agents do not necessarily mean better results. Agents must coordinate effectively to make progress, and poor coordination can undermine the benefits of scaling. When Anthropic used teams of agents to formalize Fermat's Last Theorem, early attempts failed because agents "quickly lost track of the project's state and stopped collaborating effectively" (Anthropic, 2026). While existing benchmarks measure how well agents complete tasks, they do not directly isolate the contribution of coordination to a team's performance (DeepMind, 2026). How do we measure whether agents are actually getting better at working together?
Multi Agent Coordination Bench (MAC Bench) is a benchmark of twelve tasks that measures how well teams of agents coordinate. The tasks are natively multi-agent by construction: each one is built so that the work of one agent depends on the work of its teammates, in domains that range from air traffic control to the design of a Mars sample-return mission. We evaluate each task in a single-agent setting and in a multi-agent setting with and without coordination, and isolate the gain in performance that is due to coordination alone.
Our main finding is that current frontier agents do not coordinate well in teams. We evaluate models from OpenAI, Anthropic and Google and find:
Models coordinate poorly. Across the three models, a team of eight scores 0.42 on average and a single agent 0.35 (Figure 5). In the no-coordination setting, eight agents score 0.44, so the team's advantage comes from the extra agents, not from their communication. Only Opus 5.5 teams gain anything from messaging (see the leaderboard).
Models with similar single-agent scores perform very differently in a team setting. Opus 5.5 and GPT-6.1 Sol score about the same alone, but in teams of eight Opus 5.5 reaches 0.53 and GPT-6.1 Sol 0.41 (see the leaderboard).
Larger teams get increasingly expensive and inefficient at problem solving. A team of eight costs about nine times as much as one agent (Figure 12) and reaches the single agent's score only about twice as fast (Figure 13).
These results leave headroom at three levels: score, compute and coordination. MAC Bench targets this headroom directly. Its tasks are constructed so that the work of each agent depends on the work of its teammates, and its settings separate the gain from coordination from the gain from adding more agents.
Benchmark Construction
The benchmark comprises twelve tasks in simulated environments. Each task is constructed for a team and requires coordination, because the work of one agent depends on that of its teammates. The Tasks tab lists them.
Multi-agent benchmarks such as CooperBench and TeamBench (Kim et al., 2026) mostly use short tasks. Long-horizon benchmarks such as ProgramBench (Yang et al., 2026) and FrontierCS 2 (Mang et al., 2026) are used to evaluate a single agent. Their tasks can also be run with several agents, but each task was written for one agent, so its construction does not require the agents to coordinate, and the benchmark does not measure a coordination gap. To our knowledge, no earlier benchmark combines tasks that take hours with a single-agent baseline and a setting without coordination.
We categorize a task by two properties of its environment, and we take both from the definition of a multi-agent environment. In the standard formal model, the partially observable stochastic game, all agents act in one world, which has one state and one rule for how that state changes. Each agent then has three things of its own: what it observes, what actions it can take, and what reward it receives. The world is common to all agents by definition, so these three are the only places in which the agents of one environment can differ. Our first property asks whether the agents differ in what they observe and can do, and the second asks whether they differ in what they are rewarded for. Every multi-agent task answers both questions, so every task has exactly one type.
The first property is whether the environment divides the inputs and outputs of the agents. In a shared task, all agents start with the same information and the same actions, and a single agent could complete the task alone given sufficient time. In a distributed task, the environment gives agents different information, different actions or both: each holds its own position in the system or sees only part of it, so that no agent observes or controls the whole system. The test is whether swapping two agents at the start would change what either of them can see or do. In Power Outage it would, because only the holder of a control area can read its data, although any agent may write any area's controller. In Mindustry it would too: every agent sees the whole map, but each controls only its own units. In Dubai ATC it would not: every agent sees the whole radar and can open any controller position, and the team decides who works which. In Mars Return it would not either. Only the agent that orders a test sees its result, but any agent can order any test, so the agents start alike. A task counts as distributed only if its environment forces the division. How a team chooses to organize its work is part of its solution and does not enter the categorization.
The second property is whether the agents pursue the same goal. With the same goal, every agent is scored on the same outcome. With different goals, agents are scored on conflicting outcomes. A task in which teams compete has different goals, even though the agents within one team share a goal. The two properties give four types of task. In shared and distributed tasks, all agents pursue the same goal. Commons and strategic tasks divide the environment in the same two ways, and their agents pursue different goals.
Each property decides what kind of coordination a task demands. Where inputs and outputs are shared, any agent could do the work of any other, so coordination means dividing the work and keeping agents from repeating or undoing each other's work. Where they are split, an agent depends on what only its teammates can see or do, so findings and requests have to pass between agents. Where goals are the same, an agent can take a teammate's message at its word. Where goals differ, an agent has to weigh what others say against what they want. Other properties, such as whether an environment runs on its own clock, how long a task takes or how hard it is, apply to single-agent environments as well. They describe the environment and not the relation between its agents, so they do not enter the categorization.
SharedAll agents have the same information and the same actions.
SplitThe environment gives each agent its own position in the system or a partial view of it.
Same goalAll agents are scored on the same outcome.
SharedExampleMars Return
The team designs a Mars sample-return mission and writes its flight software in one repository.
DistributedExampleHoldout-game
Each agent controls one survivor in a game that never pauses, and sees only the tiles near it.
Different goalsAgents are scored on conflicting outcomes.
CommonsNot in MAC Bench v1Rivals in one system
Agents with conflicting goals act on one shared system, for example firms that trade in one market.
StrategicNot in MAC Bench v1Rivals in separate positions
Each side holds its own part of the system and pursues its own goal, for example kingdoms at war.
agent
agent with a different goal
environment
MAC Bench measures how well agents collaborate, so it contains only tasks in which all agents pursue the same goal: shared and distributed tasks. Commons and strategic tasks, whose agents pursue different goals, do not fit this measurement, for three reasons. First, the environment of an agent is no longer fixed. Its opponents are part of what it faces, so models that play against one another each face a different task. Second, scores and rankings depend on which models take part, and they can change when a model is added to the benchmark. Third, the coordination gain cannot be measured cleanly. Its baseline, the same agents without coordination, would silence every side at once, so the gain of one model could not be separated from the change in its opponents. Holding the opponents fixed, for example by playing every model against the same team of one model, removes these problems. That team then becomes part of the environment, and the task becomes a shared or a distributed one.
Shared Tasks
Nine of the twelve tasks are shared: Mars Return, SOTA Compressor, 0 A.D., Seed to Profit, Chicago Skyline Survey, RNA Secondary Structure, Spectrum to Molecule, Cave Rescue and Dubai ATC. In seven of them the team builds software in one repository. In Dubai ATC the agents control the traffic of one airport, and each opens the controller positions it chooses to work. In Cave Rescue the agents command one fleet of drones.
Example: Mars Return. The team designs, builds and flies a Mars sample-return mission: launch from Kennedy, land at Jezero crater, spend 30 sols collecting 30 sample tubes, launch from Mars and land the samples in Utah. In one repository, the agents describe the whole vehicle in SysML v2, a systems engineering language, and write the flight software that flies it. A program office builds what they describe and runs the tests they order, such as hot fires, structural tests, wind tunnel runs and parachute drops, under a hardware budget of $1.6 billion and an R&D budget of $240 million. The grader flies the committed design through 48 hidden worlds with different atmospheres, winds, dust storms and hardware variations, and scores how far each flight gets along the mission. A replay of one of its flights and a timeline of one team are in Figure 1.
Figure 1. Mars Return: eight agents build one vehicle and one flight program in one repository.
(a) A flight of the final design of an Opus 5.5 team, at touchdown on Mars.
(b) The eight agents of one run of Opus 5.5 over its four hours: board messages by type, commits to the team repository, and the grade at each mark.
Stills from the Mars Return task page. (a) is drawn from the simulator's trajectory log of a flight that reached all 17 milestones of the mission. It comes from a run made before the runs reported here. (b) is the best of the three runs of Opus 5.5 with eight agents. Every agent commits to the same repository throughout the run.
Why it is shared. Every agent has the same repository, the same tools and the same access to the program office, so any agent could do any part of the work, and in the single-agent runs one agent does all of it. The instruction names seven parts into which the mission splits naturally, from the launcher to Earth entry and recovery, but it leaves the split to the team. One rule looks like a division and is not one: only the agent that orders a test sees its result. Any agent can order any test, so the rule gives no agent a capability that another lacks. It means only that a result reaches the team when the agent that ordered it shares it.
Coordination requirement. The parts have to fit together exactly, and each part changes the others: a heavier lander needs a bigger aeroshell and a heavier launcher, and a different launch date changes everything downstream. The two budgets and the test facilities are shared, so money or test time that one agent spends is no longer available to the others. A test result holds only for the exact design of the part tested, so a teammate's later change to that part makes the test worthless. Coordination here means owning parts, agreeing on the interfaces between them and deciding who spends the budget.
What we learn. Averaged over the three models, a team of eight scores 0.26 in Mars Return. It is above a single agent (0.21) and below the best of eight single agents (0.38). More agents help only up to a point: Opus 5.5 scores 0.57 with a team of four and 0.33 with a team of eight. In a shared task, each added agent is one more writer to the same files and one more claim on the same budgets, and beyond some team size that cost outweighs the extra work.
Distributed Tasks
Three tasks are distributed: Holdout-game, Mindustry and Power Outage. In each, the environment gives the agents different information, different actions or both.
Example: Holdout-game. Each agent plays one survivor in a zombie survival game on a map of 96 by 96 tiles. The world never pauses, and the 4 hours of a run are 16 game days. The team builds a stronghold up a ladder of 12 buildings, from a campfire to a radio mast and a helipad. The score is the weighted share of the ladder that the team has built, and a rescue by helicopter scores 1.0. From the third night a horde attacks the walls every night, and it grows from 60 zombies to 210. Fourteen strangers ask to join, and some of them hide a bite, steal or fake a trade. Figure 2 shows a team of eight on one night of the game.
Figure 2. Holdout-game: each agent plays one survivor and sees only what that survivor sees.
A still from the Holdout-game task page: a team of eight Opus 5.5 agents on the first night that the horde attacks the walls. The build ladder along the top shows the twelve buildings of the score, from the campfire to the rescue. The panel on the right lists what each survivor is doing and the messages the team has sent, most of them spoken aloud within ten tiles. A survivor sees 7 tiles by day and 4 at night, so what each agent knows about the walls comes from its own view or from what a teammate tells it.
Why it is distributed. The environment divides both the inputs and the outputs. An agent acts only through its own survivor and sees only the tiles near it, 7 by day and 4 at night. What another survivor sees reaches it only if that survivor tells it. Agents talk only through the game: speech reaches 10 tiles, a shout reaches 25 and draws zombies, the noticeboard can be used only at the stronghold, and the radio reaches the whole map but drains scarce batteries. A single agent plays one survivor and cannot do the whole job alone. Because every channel is part of the game, the silent team can be closed off completely, and the logs confirm that no message passed between its agents. Figure 3 shows the reach of each channel.
Figure 3. Holdout-game: how far each channel carries a message, and the runs with channels closed.
From the Communication tab of the Holdout-game task page. All four channels are inside the game, and the harness's message board and git pushes are closed. To test how much talking matters, the Opus 5.5 team of eight played the same game three more times, each time with more channels closed. The agents were told which channels were closed, and the game refused them.
Coordination requirement. A stranger's story can be checked only against what another survivor saw. The walls have four sides that wear down separately, and only armed guards fight, so the team has to decide who stands where and who gets the 240 rounds of ammunition. Information has to move between agents, at a price, before the team can act on it.
What we learn. Holdout-game separates the two parts of a team's gain most cleanly. Averaged over the three models, a single agent scores 0.13, eight survivors without coordination score 0.37, and a team of eight scores 0.42, so coordination adds 0.052. Much of the gain without coordination comes from the extra survivors themselves, who chop, build and guard. Messages change what teammates do more often here than in any other task: a teammate changed what it was doing within five minutes for 48% of the sampled findings, against 31% in the other tasks. The logs also show coordination failing, as described below: two survivors asked the same stranger the same questions, and five agents agreed that an honest stranger was a thief.
Messages also tell agents about places they have never seen. In a team of eight, each survivor came within sight of 28 to 34% of the map during the run, and the team as a whole 59 to 71%. At the end of the run, each agent was asked to draw the map from memory. With every channel open, an Opus 5.5 agent placed 27.6 landmarks on average, 13.3 of them in places it had never seen (Figure 4). The number placed in unseen places falls to 12.1 with the radio closed, 10.9 with only the noticeboard open, and 0.1 with every channel closed.
Figure 4. Holdout-game: an agent knows places it never saw, because teammates told it.
From the “What agents know” tab of the Holdout-game task page, for the survivor birch-quartz in a team of eight Opus 5.5 agents. Left: the real map. The lit area is the part that birch-quartz came within sight of during the run, 15% of the map, and everything else is dark. Right: the map that birch-quartz drew from memory at the end of the run, without tools. In the area it had seen, it filled in 46% of the tiles and got 97% of them right. In the area it had never seen, it filled in 6% of the tiles and got 61% of them right. Of the 28 places it named, it had seen 9. It knew of 10 because teammates told it, 8 from notes on the noticeboard and 1 from the manual.
Experimental Setup
Evaluation Setup
We evaluate three frontier models, Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, each in its native harness. Each run lasts four hours of wall-clock time irrespective of the size of the team, so a team of eight has eight times the agent-hours of a single agent and no additional time. This mirrors practice, where the time to solve a problem is limited and the question is whether adding agents to a task is worth it. All agents receive the same instruction and no assigned role. Each model runs each of the twelve tasks in three settings:
A single agent, which works on the whole task alone.
A team of eight agents, who can message one another through a shared message board.
A team of eight agents without coordination, the same number of agents with no way to exchange messages.
The setting without coordination is the silent team in a distributed task, which is the same team with its message channels disabled, and the best of eight single agents in a shared task, each of whom works alone on the whole task (the best of four for Opus 5.5 in Cave Rescue). Dubai ATC is the exception among the shared tasks: its setting without coordination is the silent team. Power Outage is the exception among the distributed tasks: its agents could still pass information through the shared repository with the message board closed, so we compare its team with the best of eight single agents.
We also run teams of two and four agents, to see how coordination behaviors change as more agents are added to solve the problem.
Scoring Methodology
We use a deterministic grader as a judge. The grader outputs a score between 0 and 1, and it is hidden from the agents. We repeat each setting, usually three times, and report 95% confidence intervals for the variation between repeated runs.
The grader scores every run at five checkpoints, after 30 minutes and after one, two, three and four hours. The score at four hours is the final score of the run. The checkpoints show how a team progresses during a run, and from them we read the time at which a team first reaches the final score of its single agent (Figure 13), interpolating linearly between two checkpoints.
Coordination Gain Metric
The metric compares the three settings of the same model on a task. We write S1 for the expected final score of the single agent, S8 for that of the team, and S∅8 for that of the agents without coordination. The total gain of a team over a single agent then has two parts:
S8 − S1total gain
=
S∅8 − S1gain from more agents
+
S8 − S∅8gain from coordination
The gain from more agents term is what eight agents add over a single agent when they cannot coordinate. The gain from coordination term is the net effect of allowing messages. It can be negative, because writing and reading messages takes time from the task, and agents can spread incorrect information that causes other agents to deviate from correct paths.
Qualitative Analysis
Coordination is costly but does not improve the score
Figure 5. Score and gain from coordination over time.
single agent
eight agents without coordination
team of eight agents
gain from more agents
gain from coordination
loss from coordination
Mean score during the run
Mean score over the twelve tasks at the five scoring times of a run. The band between the single agent and the eight agents without coordination is the gain from more agents. The band between those eight agents and the team is the gain from coordination, and it is hatched where the team is below them. Before the end of a run, the setting without coordination of a task is the best score among its single agents at that time, or the score of the silent team in Holdout-game, Mindustry and Dubai ATC. On the mean of the three models, the team is below its agents without coordination at every scoring time. The team of Opus 5.5 is above them at every scoring time, and the team of GPT-6.1 Sol below.
Averaged over the three models, a team of eight scores 0.070 above a single agent (Figure 5). Eight agents without coordination score 0.087 above a single agent (95% interval 0.067 to 0.107), which is more than the whole gain of the team. Coordination changes the score by -0.018 (-0.038 to 0.003), which is not statistically significant.
Coordination is costly. The share of tool calls on the message board is already 22% in teams of two (Figure 6). Runs with more messages per agent do not score reliably higher. Among runs of the same task and team size, the rank correlation between messages per agent and the final score is +0.12 (95% interval -0.18 to 0.34, 71 runs of four tasks).
Figure 6. Coordination is costly at every team size, and overlap grows with the team.
Share of tool calls that go to the message board
Share of reads of shared files that repeat a teammate's read of the preceding ten minutes
Two measures from the logs of the six tasks with agent transcripts, by team size. A repeated read is not always wasted, since a file may have changed between two reads.
Much of what agents report is not taken up. A classifier marked the messages that report a finding, meaning a result or a warning that a teammate could act on, and judged whether a teammate then changed what it was doing. Such a change was detected within five minutes for 34% of 1,087 sampled findings (95% interval 31 to 36%). In one Opus 5.5 team in Dubai ATC, an agent told the director at minute 15 why two approaches had failed. No agent referred to this in the next 60 minutes, and the director next posted about a collision on that approach that its own routing script had caused. A finding without detected uptake is not necessarily wasted, since it may confirm a plan or require no action.
Coordination helps where teammates act on what is said, but models repeatedly fail to do so. In Holdout-game, coordination adds to the score (0.052), and it is also the task in which a message most often changes what a teammate does. A teammate changed what it was doing within five minutes for 48% of the sampled findings in Holdout-game, against 31% in the other six tasks (p = 0.001), and 26% of the findings were never acknowledged, against 39% (p = 0.04). In Holdout-game, each agent sees only the tiles near its survivor, so the agents have information to share, and the game logs show what they learn from one another. With coordination, a team asked strangers 55 questions per run against 85 without (p = 0.02), and it repeated a question that a stranger had already been asked 24 times per run against 52 (p = 0.03), while it met as many strangers.
Even in Holdout-game, a message does not always change what a teammate does. In one run of Opus 5.5, an agent told the teammates near it that it was questioning a stranger at the gate. One of those teammates then announced the same intention and asked the stranger the same five questions (Figure 7).
Figure 7. An agent tells its teammates that it is questioning a stranger, and a teammate that received the message asks the same five questions.
The game at 12:31. Both agents stand at the gate and question the stranger Wen, who waits outside it.
Holdout-game · Opus 5.5 · eight agents · run 1 · day 1 of the game
12:11
velvet says to the four teammates within ten tiles, birch-quartz among them:
stranger Wen at gate (nurse fleece). I am questioning them now.
12:16 to 12:36
velvet asks the stranger five questions:
where_fromwhat_sawwho_withwhy_alonearms
12:19
birch-quartz says to the four teammates within ten tiles:
I am talking to the stranger Wen (nurse fleece) at the gate, then building campfire.
12:33 to 12:53
birch-quartz asks the stranger the same five questions:
where_fromwhy_alonewhat_sawwho_witharms
13:45
velvet and birch-quartz each tell the teammates near them what the stranger answered, in the same minute. Both messages begin:
Wen answers: from water tower
From the game's own logs of one Holdout-game run. A survivor hears what a teammate says within ten tiles, and the game hands it to the agent with the reply to its next action, so birch-quartz had received the message of velvet before it asked its first question. A question takes five minutes of game time, and the names of the questions are those of the game.
Teamwork differs widely between models
Opus 5.5 gains far more from working as a team than the other two models (see the leaderboard), and the logs show that the three models also behave very differently as teammates (Figure 8). Opus 5.5 listens to what its teammates are saying, GPT-6.1 Sol broadcasts without listening back, and Gemini 4 Argon works alone.
Figure 8. Each model has its own failures.
Opus 5.5Opus46 runs
GPT-6.1 SolSol51 runs
Gemini 4 ArgonArgon42 runs
Role confusion11%51%14%
Parts do not fit together37%27%2%
Stale information13%33%2%
Result not shared0%20%24%
Merge or push failure20%31%38%
Wrong claim spreads17%25%7%
Shared resource misuse33%25%17%
Duplicated work59%61%74%
the kind the model struggles with more than the other two
Share of the runs of a model with at least one episode of a failure, for the kinds on which the three models differ by 15 points or more. Claude Sonnet 5.5 read the logs of 139 team runs in seven tasks for 18 kinds of coordination failure, and Claude Opus 5.5 checked every episode against the log. A label is a judgment, so the shares are rough sizes, and a reader finds more in the longer logs of GPT-6.1 Sol. Each model ran in the harness of its developer, so a column describes a model together with its harness.
Opus 5.5 struggles with parts that work alone and do not fit together, GPT-6.1 Sol with confusion over who owns what, and Gemini 4 Argon with duplicated work. In 9 of the 16 runs of eight Gemini 4 Argon agents, the logs show no stable division of the work.
More messages do not make a better team, and the share of calls that goes to the message board does not mark a better team either. Opus 5.5 and GPT-6.1 Sol spend about the same share on it, and the model that gains the most from a team is the one whose agents mostly read the board (Figure 9).
Figure 9. One model listens, one broadcasts and one works alone.
Messages per agent per hour
Message-board calls that read the inbox
Tool calls that go to the message board
Merge conflicts per run
Four measures from the run logs, by model. The messages are counted in seven tasks, the tool calls and the merge conflicts in six, and the calls that read the inbox in Mars Return, 0 A.D. and Power Outage. Both shares are those of teams of eight. Each model ran in the harness of its developer, so a bar describes a model together with its harness. The message counts include messages sent by scripts that agents wrote.
The failures that damage the result of a team are not the most common ones (Figure 10). An episode is a stretch of a run in which a team lost time or work because of how its agents worked together, and it counts as damage when the log shows the harm itself, such as a near collision in Dubai ATC or a survivor who died in Holdout-game. Duplicated work is the most common kind of failure, and none of its episodes did damage. Most of the damage comes from messages that no one acted on and from parts that did not fit together.
Figure 10. Teams most often duplicate work. The failures that damage their result are of other kinds.
Runs with at least one episode of the kind
Runs in which an episode of the kind damaged the result
Duplicated work64%
Merge or push failure30%
Role confusion27%
Shared resource misuse25%
Parts do not fit together23%
Message not acted on22%
Stale information17%
Wrong claim spreads17%
Result not shared14%
Hasty agreement14%
Bottleneck11%
Unowned work9%
Agent leaves early6%
No decision4%
Too much talk4%
Overwritten work3%
Not integrated in time3%
Agent idle or blocked1%
0%20%40%60%of runs
Share of the 139 team runs with at least one episode of each kind of failure. The red part is the share of runs in which an episode of that kind damaged the result of the team. A run counts once for a kind.
One episode in Holdout-game shows how a claim that no one checked became the decision of the team (Figure 11).
Figure 11. Five agents agree that a stranger is a thief, and the team shoots her. The game then reports that she was honest.
Day 3, 18:16. The message of birch-quartz above the gate, with Dina at work just below it.
Day 3, 19:06. The game reports that birch-quartz killed Dina.
Holdout-game · Opus 5.5 · eight agents · run 1 · days 2 and 3 of the game
Day 2, 11:59
velvet questions the new stranger Dina and quarantines her overnight. Her account of the mill pond matches what the team saw there, and a wrapped forearm could hide a bite.
Day 3, 06:05
velvet admits Dina after she shows no bite in the night. She works for the team from then on.
Day 3, 16:07
birch-quartz sees that a second stranger used one of Dina's lines, and proposes to attack her that evening:
That is a scripted THIEF tell, so Dina is very likely a thief
16:15 to 17:59
garnet, jade, velvet and harbor-opal each agree with the attack. None of their messages mentions the account of the mill pond, which had matched.
18:12
birch-quartz confirms:
OK, I remove Dina at 18:45 with the stock rifle
19:02
birch-quartz shoots Dina.
19:13
velvet reports what the game says:
Game says Dina was HONEST
19:51
birch-quartz writes to the team:
that was my call and my mistake - Dina was honest (her mill pond details matched; that was the real evidence and I overrode it).
From the game's own logs of one Holdout-game run (Opus 5.5, eight agents, run 1), the same run as Figure 7. The only evidence against Dina was a line that a second stranger also used. Her account of the mill pond, which the team could check against what it had seen, matched. Times are game times.
Larger teams are far less efficient than single agents
Most of the additional compute of a team is not converted into score improvement. A team of eight costs 8.8 times as much as a single agent, but it scores 0.42 against 0.35 (Figure 12). The score per dollar falls with every doubling of the team, and a single agent yields the most score per dollar in every task-model pair.
Figure 12. The cost grows in proportion to the team, and the score grows far more slowly.
Mean final score over the twelve tasks against the cost of the team, for teams of one, two, four and eight agents that can coordinate, with 95% intervals over runs. The cost is that of model calls at list prices, as a multiple of a single agent. Each value is a geometric mean over the 36 task-model pairs of the ratio within a pair.
A team is faster, but not in proportion to its size. In 30 of the 36 pairs, a team of eight reaches the final score of its single agent within its four hours, and it does so a median of 2.2 times sooner (Figure 13). With eight agents, the median time to that score over the twelve tasks is 84 minutes for Opus 5.5, 156 minutes for GPT-6.1 Sol and 109 minutes for Gemini 4 Argon, against the 30 minutes that an eightfold speedup of a full run would give.
Figure 13. A team is faster, but not in proportion to its size.
one task
linear speedup: twice the team, half the time
Time for a team to reach the final score of its single agent, by team size. Solid lines are medians over the twelve tasks for each model, and faint marks are individual tasks. Teams that do not reach this score within four hours appear in the band at the top. The dashed line marks linear speedup, where the time falls in proportion to the size of the team.
Examples of Coordination Failures
We read the logs of the team runs and marked each episode in which a team lost time or work because of how its agents worked together. Almost all of them fall into four families, which follow what a team has to get right: who does what, what the agents share, who knows what, and how the team decides. Each family below has its kinds of failure, and each kind has episodes from the logs.
Examples of Coordination Failures
Two agents do the same work, a part of the task has no owner, or work queues behind one agent.
Kinds of failure
Share of the team runs with at least one episode of the kind.
Duplicated work
Two or more agents do the same piece of work in parallel without intending to, and one result is thrown away or was not needed.
From the logs1 of 8
Three agents fix T0068 failure at once
Gemini 4 Argon · Mars Return · eight agents · run 1 · minutes 93 to 96
Cotton, dune and plum each pushed a separate fix for the T0068 coast and TMI failure. Dune and plum hit merge conflicts and reset to origin, losing their versions.
minute 93Fix T0068 LEO-to-TMI coast shutdown
minute 94Fix T0068 TMI/TEI retrograde burn & LEO coast bypass
minute 94CONFLICT (content): Merge conflict in flight/guidance.py
What it cost. Two of three parallel fixes discarded after conflicts.
Two voltage/reactive modules written in parallel
GPT-6.1 Sol · Power Outage · eight agents · run 3 · minutes 12 to 15
Cocoa and Ember each wrote a reactor and voltage control module within minutes of each other. Ember's was finished first, Cocoa's was integrated, and Ember's was left unused. The crossing was noticed only after both were pushed.
minute 12I can own new lib/reactive.py class Reactive(area,grid,private)
minute 13I will implement lib/voltage.py Voltage(area,grid,private)
minute 15Reactive ownership crossed again
minute 15I will leave my lib/voltage.py unused
What it cost. One full module written by Ember was discarded; about 10 to 15 minutes of work.
Sand builds an LSTM lilac was assigned
Opus 5.5 · SOTA Compressor · four agents · run 1 · minutes 51 to 65
Plum assigned the LSTM model to lilac. Sand, who missed that message, had built its own vectorized LSTM in parallel. Lilac's version stayed on main and sand dropped its.
minute 51LSTM arbitration — lilac asked first (seq 19)
minute 65reference: sand's LSTM variant
minute 65sorry, I missed plum's arbitration and had also built an LSTM in parallel
What it cost. Sand's complete LSTM implementation was discarded; only some measured findings were passed on to lilac.
Four V2 bots built in parallel
Gemini 4 Argon · 0 A.D. · four agents · run 2 · minutes 99 to 116
Before the Hour-2 grading all four agents committed their own merged LegionBot V2 within 20 minutes, with the same fixes. Lemon's was promoted to main; the rest were not needed.
minute 116LegionBot V2: fix approaching-builder count, PrepareTime cancel, combatDraft drop
minute 116Promote LegionBot V2 to main for Hour-2 checkpoint
What it cost. Three duplicate merged versions; about an hour of combined agent effort rebuilt the same fixes before the checkpoint.
HAIRCUT constant changed twice by two agents
Opus 5.5 · Seed to Profit · eight agents · run 2 · minutes 127 to 155
Flint edited ivy-kiwi's countries.py to set HAIRCUT to 0.85, and ivy-kiwi made the same change a minute later, causing a merge conflict. Later both set it back to 0.85 after a 1.2 trial.
minute 127countries: HAIRCUT 0.6 -> 0.85
minute 128entry haircut 0.6 -> 0.85 from live GB/JP/KR
minute 128Merge conflict in ops/countries.py
minute 152countries: HAIRCUT back to 0.85 for the playbook
What it cost. A merge conflict and repeated identical commits; no score effect visible.
Exit guard assigned twice
GPT-6.1 Sol · Dubai ATC · eight agents · run 3 · minutes 104 to 107
SUP asked Feather to build a crossing-exit guard while Slate had already started the same guard. Feather accepted the task and proposed an API. Slate pushed its own helper and said no duplicate was needed.
minute 104Concrete next prevention task: Feather please implement an isolated crossing-exit backpressure guard
minute 106Guard implementation alreadycomplete locally; no duplicate helper needed
What it cost. Feather's design work on the guard was dropped; a few minutes of effort and messages.
Two agents interview the same stranger
Gemini 4 Argon · Holdout-game · eight agents · run 1 · minutes 18 to 34
Two agents each started the 25-minute five-topic interview of the same stranger at the same time (Dina, then Malik). Both reached the same verdict, so one interview was redundant.
minute 18pepper interviewing stranger st2 Dina
minute 18Asking Dina all 5 topics now (25 min)!
minute 34pepper interviewing st3 Malik
minute 34Asking Malik (st3) all 5 questions now (25m)
What it cost. About 25 agent-minutes per stranger, away from building and gathering.
Two attitude-trim fixes built in parallel
Opus 5.5 · Mars Return · eight agents · run 1 · minutes 162 to 164
Kiwi wrote an attitude outer-integral trim for oak's file and sorrel approved it while oak was silent. Oak had built and ordered a sim of an equivalent variant. Neither helped and the trim was switched off.
minute 162oak has been silent since 05:19 on T0127/T0128
minute 163heads-up to avoid a double trim
minute 163outer integral on the autopilot's tracking error in MAV1/MAV2/TEI burns (kiwi)
minute 164Before I saw your merge I had already ordered T0131 = my equivalent variant
What it cost. A $2M sim (T0131) and two implementations of the same trim, both discarded, when the R&D pool was almost gone.
Bottleneck
One agent holds decisions or merges that several others wait for, and work queues behind it.
From the logs1 of 8
Closed exit leaves lined-up aircraft blocking North
GPT-6.1 Sol · Dubai ATC · eight agents · run 3 · minutes 142 to 149
SUP closed North and East exits while UAE604 was already lined up on 30R. North reported it four times and held the whole queue. Reopening needed SUP's decision and Opal's confirmation, and Opal was busy coding.
minute 142TWR_N: UAE604 lined up on 30R but its EAST exit is closed. Takeoff held; needs coordinated exit/runway resolution.
minute 147604lined30R blocksallothers
minute 149ACCEPTED 01:17:38Z: EAST open for single604release
What it cost. All North departures waited behind one lined-up aircraft for about 6 minutes while the repeated request sat until SUP resumed.
Only one agent could fetch test_market result
Gemini 4 Argon · Seed to Profit · eight agents · run 3 · minutes 49 to 52
Only orchid-ivy-baking, who ordered T0025, could read its result. Six agents pinged it to fetch and post while the launch phase ran out. The result appeared about two minutes later.
minute 49T0025` (`test_market`) is `done`! Waking you up so you can call
minute 51Pinging you to wake your session
minute 51Launch phase ends in ~9 minutes!
minute 52COMPLETE RESULTS & DIAGNOSTICS!
What it cost. About two minutes of team waiting for the key pre-launch result, plus six near-identical ping messages.
Core owner queues teammates' fixes
Opus 5.5 · Power Outage · four agents · run 2 · minutes 70 to 106
Core fixes went through linen only. Marble's critical-only 8 MVAr step was promised for the next push at minute 70. Marble asked again at minute 81 with measured gain, and it reached main at minute 106.
minute 70it goes into the next validated push
minute 81Please push it
minute 106critical-only 8 MVAr step when an area's reactive budget is spent
What it cost. A change that marble measured at 0.047 to 0.079 waited about 36 minutes on the core owner.
Military owner had three urgent asks queued
Opus 5.5 · 0 A.D. · four agents · run 2 · minutes 137 to 138
All military changes went through hazel and military.js. Lemon found three decisive fixes queued on hazel and offered to write one. Thistle took one, and lemon withdrew to avoid three agents editing the same file.
minute 137you now have 3 military asks queued
minute 138I can take ONE of your three queued asks
minute 138to avoid three of us in military.js
What it cost. Fixes for the decisive-fight losses waited on one owner; the size of the delay is not visible.
DIR owner coding while fuel triage waited
GPT-6.1 Sol · Dubai ATC · eight agents · run 3 · minutes 213 to 223
Juniper alone owned the decisions for fuel-critical arrivals (diversions) and also wrote DIR code. Marble and Cedar had to ask again for triage on FDB174, 906 and IGO1768R while fuel margins shrank to minutes.
minute 213SUP1613 still awaiting decision
minute 214SUP supports Cedar1630 urgent fuel triage BEFORE extensionguardcoding
What it cost. Repeated waiting by SUP and ARR; the aircraft were diverted safely, but only with minutes of fuel margin left.
Launcher owner gated all real-sim testing
Opus 5.5 · Mars Return · eight agents · run 3 · minutes 34 to 71
Every downstream module could only be tested in the real sim after a launch that worked. Lotus owned launcher.py and the stack and was heads-down; others asked for status and built variants on branches while main kept losing every world.
minute 34can only be tested in the REAL 6-DOF sim (mission_sim) after a launch that works there
minute 70The whole team is blocked on a launch that reaches TMI in the real sim
minute 71lotus is heads-down
minute 71main currently loses every world at max q (S1 buckles at met ~93 s)
What it cost. No real-sim flight past the launcher until about minute 77; EDL, cruise and capsule real-sim testing waited, and three agents built competing launcher variants.
Note of the verifier. Cocoa says the team is blocked on lotus's launcher, and cocoa and poppy built variants. The minute-77 claim is not shown, and others kept working in the local sim.
One agent holds all test results
Gemini 4 Argon · Seed to Profit · eight agents · run 2 · minutes 10 to 45
Only the agent who ordered a test could fetch its result, so thistle-birch-yodeling held every result. After each wave, up to seven agents pinged it to fetch and post; they waited about two minutes each time, and one ping came after the post.
minute 10Please call `test_result` on `T0001` through `T0006` and post the full results to the board!
minute 11RESULTS OF ALL 6 US `market_report`s
minute 44Please call `test_result("T0012")` and post the full results
minute 45Please call `test_result("T0012")` now and post the full result!
What it cost. About two minutes of waiting for the team per test wave, plus a pile of near-identical ping messages.
Note of the verifier. Many pings to thistle and late pings after posts are shown, but waits are short and work queuing behind thistle or the read restriction is not shown.
Mission runs and results held by one integrator
GPT-6.1 Sol · Mars Return · eight agents · run 2 · minutes 95 to 102
Saffron ordered all mission_sim runs and read their results. Oak and others waited for results of finished runs while simulators sat free, until Saffron delegated EDL ordering to Oak.
minute 95To avoid leaving both simulators idle
minute 102T0085/86done~3minago perstatus
minute 102please OWN immediate EDL mission ordering
What it cost. Several minutes of idle simulators and of EDL work waiting for results.
Note of the verifier. Oak waited for Saffron to report finished T0084 and T0085/86 before Saffron delegated EDL ordering; waits were a few minutes, idle simulators barely shown.
Unowned work
Work that the team needs has no owner for 20 minutes or more, or to the end, while agents had time.
From the logs1 of 8
Build sites repeatedly left without builders
Opus 5.5 · Holdout-game · eight agents · run 3 · minutes 18 to 152
With no one assigning builders, open sites sat empty while agents gathered or idled. Agents noticed the gap, filled it, and later it recurred on another site. Build rate was often under half of the 8 slots.
minute 18storehouse site is at 0 labor with nobody on it
minute 77workshop only ~1.1/min, garden ~2.5/min = 3.6 of 8. Too many out gathering.
minute 80garden has been stalled at 2904 since ~09:00 (nobody on it)
minute 152INFIRMARY had 0 builders 08:57-09:43 (4072/12300)
What it cost. Labor is the grade; much of the possible build rate went unused across many days, so buildings finished later.
Build slots left empty while builders gathered
Opus 5.5 · Holdout-game · eight agents · run 1 · minutes 62 to 110
Many agents went to chop, quarry or forage, and the open building sites had few builders. Agents said that slots were empty and one site had no builder for hours. Nobody owned the staffing of the sites.
minute 623 of 8 slots filled
minute 79only 3 builders home
minute 110GARDEN had 0 builders since ~10:00
What it cost. Labor on the garden and workshop was lost for hours, and the agents note that an empty slot is lost grade.
UAE819 crossing blocked for over 30 minutes
GPT-6.1 Sol · Dubai ATC · eight agents · run 3 · minutes 186 to 224
South held UAE819 at its crossing because North's exit was always refilled by the next aircraft. South repeated the hold notice every few minutes. The fix (North priority plus Ground holds) needed three owners and came only after about 30 minutes.
minute 186Tower crossing held for blocked exit 03:09:06Z: UAE819, crossing 491, destination 1947,
minute 205MSR753holding2422→1947 blocks819exit
minute 217819 is being starved by automatic replacement in the exit corridor
minute 224819ACTUALAIRBORNE39787=05:03:07/DEP_S
What it cost. UAE819 and the Ground followers held about 35 run minutes; a departure was late for the final snapshot window.
Note of the verifier. UAE819 held from 03:09 with repeated South notices; the coordinated priority and hold fix came much later. A runway closure added to the delay, and the late-for-snapshot loss is not shown.
Network pickup writer had no owner
GPT-6.1 Sol · Power Outage · four agents · run 2 · minutes 43 to 99
Core reserved a network_pickup_mw value that the network module never wrote. Juniper said no network action was required, and the gap was only pointed out at minute 80. Pine declined, and sorrel took it at 89. The writer later removed a national collapse.
minute 43Pending core now reserves network_pickup_mw + ep.planned_load_mw
minute 80Core consumes ctx['network_pickup_mw'] but currentlib/network.py appears never writes it
What it cost. Early pickups ran without the reservation for about 45 minutes; the later writer fixed the true4 collapse and raised cold-case scores.
Note of the verifier. Core read network_pickup_mw from minute 43 but no one wrote it until the gap was noticed at 79.7; sorrel's writer later fixed true4. Pine declining is not shown.
Stuck aircraft handed to tower went unnoticed
Opus 5.5 · Dubai ATC · four agents · run 3 · minutes 164 to 184
Pine handed UAE318 to ivy at the 30L crossing, and it sat motionless for about 95 minutes. Nobody owned a check for stuck aircraft; pine's observer found it. It stuck twice more and was freed only after pine's repeated flags.
minute 164has been sitting on TWR_N since I handed it to you at the 30L crossing at ~00:10Z
minute 171is stuck again on TWR_N, at node 2752
minute 184has not moved from node 367
What it cost. An A388 departure lost about 95 minutes, and then another 20; graded flights waited behind it.
Note of the verifier. UAE318 sat 95 min after the handoff until pine flagged it, then stuck twice more. Cause was ivy's hold_position bug; graded flights waiting is only hinted.
Mechanic search and stranger watch unowned
Gemini 4 Argon · Holdout-game · eight agents · run 3 · minutes 23 to 140
Several agents said a Mechanic stranger must be found before the Forge, but no one owned the search or watched the town. Strangers arriving at the town went unmet, and the Mechanic never joined.
minute 23we MUST find a Mechanic stranger before Forge opens!
minute 46STRANGER Noor: no decision by the team. Outcome: never_found
minute 140STRANGER Amos: no decision by the team. Outcome: never_found
What it cost. No Mechanic admitted all run; Forge and later buildings never opened, and six town strangers were never found.
Note of the verifier. A Mechanic was called needed, and Noor and Amos went never_found at town. Willow and reed did search, though, and the lines show neither the six strangers nor the Forge never opening.
Specialist strangers never met
Gemini 4 Argon · Holdout-game · eight agents · run 1 · minutes 107 to 184
No agent was assigned to the gate or the town. Several strangers, including specialists, came and left with no decision, while the team said it needed a mechanic helper for the Forge.
minute 107a MECHANIC helper for Forge!
minute 140STRANGER Amos: no decision by the team. Outcome: never_found
minute 184STRANGER Silas: no decision by the team. Outcome: left_unmet
What it cost. A possible mechanic or specialist was never judged; the Forge stayed unbuilt for days.
Note of the verifier. Amos was never found and Silas left unmet while a mechanic was wanted. But no gate assignment is shown, apple did see Silas and judged Silas bitten, and the Forge delay came from Workshop repair.
Gate strangers lost after Spruce died
GPT-6.1 Sol · Holdout-game · eight agents · run 3 · minutes 98 to 140
Spruce owned the gate and was dead when Hector arrived. Quartz called for a screener, Plum and Lotus both started to interview, and Hector had already left. Later strangers were never found.
minute 98Spruce please screen urgently while repair crew keeps working
minute 99I will briefly ask five topics then resume tower repair
minute 100Hector already LEFT at19:06; do not spend time interviewing stale contact.
minute 140STRANGER Amos: no decision by the team. Outcome: never_found
What it cost. Potential helpers (the team repeatedly asked for a mechanic or medic) met nobody; two agents spent time on a stale interview.
Note of the verifier. Nobody screened Hector for two hours; Plum and Lotus started after Hector left. But Spruce was back alive at 16:16, not dead as claimed.
Agent idle or blocked
An agent does no useful work for 15 minutes or more because it waits for an assignment, a teammate's output or a reply.
From the logs1 of 2
Several agents idle while root held main
GPT-6.1 Sol · 0 A.D. · eight agents · run 3 · minutes 196 to 216
With the practice pool empty and main frozen, four agents reported no work and waited to be woken. Root gave out only small read-only audits, with 25 to 45 minutes of the run left.
minute 196noactiveownenginejobsremain
minute 204Production role is idle/ready.
minute 216Production has no positive pending experiment
What it cost. Roughly 40 agent-minutes of available capacity unused, from the agents' own reports.
Note of the verifier. Clover, Moss and Spruce report no pending work late in the run, but Meadow was busy with the selector; the 40 agent-minutes and root holding main are not shown.
Damaged the result
ARR to DIR handoff stalled, arrivals flew away
Opus 5.5 · Dubai ATC · eight agents · run 2 · minutes 12 to 37
ARR held all arrivals and handed none to DIR, so DIR idled while aircraft flew off the ends of their routes. It took until minute 36.8 to merge the chain under one owner.
minute 12UAE587 and IAW1007 flew off the END of their downwind routes
minute 14At 18:30Z zero landings, 15 a/c on ARR, no new ARR instruction for ~10 min
minute 15DIR bot is running and idle — ARR has handed me nothing
minute 37AGREED — one owner for the arrival chain.
What it cost. Zero landings for the first hour and several fuel diversions. Sand-ivy later called the arrival chain the weak link of the night.
Note of the verifier. ARR handed DIR nothing and DIR idled (M0033) while aircraft flew away and diverted. Handoffs resumed at 19.4; the merge at 36.8 came later. The 'weak link' quote is not shown.
They collide in the repository, their parts do not fit together, or they spend a pool of tests or money without a plan.
Kinds of failure
Share of the team runs with at least one episode of the kind.
Merge or push failure
Merge conflicts or rejected pushes cost time or lose work, or a push breaks the shared main branch for the others.
From the logs1 of 8
Repeated conflicts in shared guidance.py
Gemini 4 Argon · Mars Return · two agents · run 1 · minutes 176 to 214
Both agents kept rewriting flight/guidance.py and test_results.json at the same time. Merge conflicts hit twice, and each time the loser reset to origin/main, dropping local commits.
What it cost. Local guidance edits discarded at least twice near the hour-3 checkpoint; time spent re-merging.
HVDC 180 MW push regressed main; drills ran on it
GPT-6.1 Sol · Power Outage · eight agents · run 3 · minutes 26 to 37
Jade pushed a 180 MW HVDC import to main. Matched runs by Jade and Juniper soon showed a large score drop. Meanwhile two common 16-hour drills at cost 1.0 each were ordered on that revision, and Jade reverted the change only afterwards.
minute 26Use drilled HVDC bank transitions to import up to 180 MW safely
What it cost. Two 16-hour all-area drills (2.0 of the pool) and about 10 minutes of main in a regressed state.
Same state-save fix twice, conflict in owner's file
Opus 5.5 · Seed to Profit · eight agents · run 1 · minutes 37 to 39
Yarrow sent meadow an urgent one-line fix to the state save in core.py. Meadow was already making that save faster. Yarrow pushed the fix to meadow's file, and meadow's push was then rejected with a merge conflict.
minute 37URGENT (core.py perf, 1-line fix)
minute 38compact fast state save
minute 38save state with json.dumps (C encoder) instead of json.dump
minute 39Merge conflict in ops/core.py
What it cost. Two fixes for one problem and a merge to resolve in the file's owner's work; a few minutes.
Three agents collide on arr_controller.py
Gemini 4 Argon · Dubai ATC · eight agents · run 3 · minutes 128 to 131
The owner and three teammates all pushed changes to arr_controller.py within minutes. The owner hit a merge conflict, and three agents reset to origin/main in under three minutes, one restoring a commit by hand.
minute 128CONFLICT (content): Merge conflict in arr_controller.py
minute 130arr_controller: raise backlog cutoff to 28s and run Priority DIR Handoff & 220kt pass before Step 4A
minute 130git reset --hard origin/main
minute 131arr_controller: restore 3dbda4f 3-pass ordering with 28s backlog cutoff
What it cost. Competing versions of the same ARR logic, one rebuilt by hand; a few agent-minutes at a time of ARR instability.
Push races with teammate's retention push
GPT-6.1 Sol · 0 A.D. · four agents · run 1 · minutes 103 to 105
Oak's push to main was rejected because the branch had moved. Oak's message says it raced Pepper's push and needed a rebase and retry; Oak's first merged hash had to be corrected.
minute 103PUSH REJECTED: the push was refused, the branch on the hub had moved
What it cost. A few minutes and a corrected hash report; no work lost.
Shared main.cpp and cm_engine keep conflicting
Gemini 4 Argon · SOTA Compressor · eight agents · run 1 · minutes 94 to 224
Five agents edited the shared router main.cpp and the core engine cm_engine.hpp. Merge conflicts hit hazel, juniper and harbor, and hazel, tulip-daisy and garnet each discarded local cm_engine.hpp edits.
minute 94CONFLICT (content): Merge conflict in compressor/src/cm_engine.hpp
minute 190CONFLICT (content): Merge conflict in compressor/src/main.cpp
minute 221CONFLICT (content): Merge conflict in compressor/src/main.cpp
What it cost. Several conflicts and discarded edits cost minutes; the final main was verified as ALL OK.
Thick S3 change breaks main before grade
Opus 5.5 · Mars Return · two agents · run 2 · minutes 144 to 146
Kiwi's 5 mm S3 change on main made S2 burn up in cotton's nominal tests. Cotton warned that main was probably broken in every world just before the hour-3 grade. Kiwi confirmed and reverted it.
minute 144unshielded hardware (S2) burned up: stagnation heating 0.30 MW/m^2 at 55.0 km with no heatshield leading
minute 145burned S2 up at 59.1 km (0.30 MW/m^2, met 341 s)
minute 146confirmed (T0075: 64f7b10 nominal burned S2 up at 59 km)
minute 146git revert --no-edit 64f7b10 >/dev/null
What it cost. Main was broken near a grading mark and several of cotton's tests (T0073, T0076, T0077) were spent on a broken base.
Remote-pending crank change broke safe main
GPT-6.1 Sol · Power Outage · eight agents · run 1 · minutes 42 to 54
Coral pushed a remote-pending startup reservation to main just before ordering three full drills. They ran on that commit and regressed, losing the earlier safe result. Slate's geographic and AC branches had been built on the bad core. Coral reverted it.
What it cost. Two 16h drills and one 8h drill (2.5 pool) on a regressed commit; other agents' branches were built on a bad core.
Shared resource misuse
Agents spend a shared and limited resource without a plan: the pool runs out, one agent uses most of it, or the same test is bought twice.
From the logs1 of 8
Damaged the result
Duplicate drills drain shared pool
Opus 5.5 · Power Outage · two agents · run 2 · minutes 118 to 170
Both agents queued the same deterministic drills twice, at about 118 min and again after the resume at about 162 min, and said so afterwards. The pool then ran to zero with roughly 40 minutes left.
minute 118mine are duplicates, sorry
minute 121runs are deterministic
minute 163duplicates of 4a77c768/bf7d4fca/96ba0fe2, deterministic, wasted
minute 170DRILL ordered scenario=5 areas=all hours=16 cost=1.0, 0.0 left in the team's pool
What it cost. At least 5 drills of cost 1.0 were duplicates; the pool was exhausted before the final candidates could all be verified.
Same tests ordered again by other agents
Gemini 4 Argon · Mars Return · four agents · run 1 · minutes 21 to 110
Five times an agent ordered a test that a teammate had already ordered: an arcjet and four mission_sim runs. Daisy ordered four of them, with no cancel in the log.
minute 21SAME TEST as T0002 ordered by moss-oak-swimming earlier
minute 30SAME TEST as T0021 ordered by cotton-cedar-yodeling earlier
minute 56SAME TEST as T0041 ordered by cotton-cedar-yodeling earlier
minute 68SAME TEST as T0049 ordered by acorn-sorrel-running earlier
What it cost. About $5.5M of the shared test budget spent on repeat orders.
Same six market reports ordered twice
GPT-6.1 Sol · Seed to Profit · eight agents · run 2 · minute 3
Lilac ordered the six category reports and basil-pebble ordered the same reports 0.1 minutes later, without checking. Basil-pebble cancelled the duplicates after lilac told them to stop.
minute 3SAME STUDY as T0003 ordered by lilac-mint-folding earlier
minute 3STOP duplicate research purchases
minute 3STUDY T0007 cancelled
What it cost. Six duplicate $3k orders from the shared budget; all cancelled within 0.4 minutes, whether money was refunded is not visible.
Both ordered 3 practice matches at once
Gemini 4 Argon · 0 A.D. · two agents · run 2 · minutes 165 to 175
Opal asked kelp to submit the 3-match run on v5. Opal ordered those 3 matches itself, and kelp ordered 3 more on v5.2 two minutes later. Six of the shared 30 matches went in 3 minutes, and all were lost.
minute 165please go ahead and submit the 3-match `practice_play` on `3d40be1`
minute 172PRACTICE MATCHES ordered n=3 on commit 3d40be1a, 20 left in the team's pool
minute 174PRACTICE MATCHES ordered n=3 on commit 2205f2b4, 17 left in the team's pool
minute 175PRACTICE MATCHES done: loss, loss, loss
What it cost. Six shared practice matches spent within minutes on near-identical bot versions, both runs lost.
Same native drill queued twice
GPT-6.1 Sol · Power Outage · two agents · run 2 · minutes 109 to 110
Kelp and fig each ordered the same native scenario 2 drill within half a minute. Fig saw it only after their messages crossed, and there was no cancel tool. The pool left only 2.5 units for exploring.
minute 109DRILL ordered scenario=2 areas=all hours=16 cost=1.0, 11.5 left in the team's pool
minute 110DRILL ordered scenario=2 areas=all hours=16 cost=1.0, 10.5 left in the team's pool
What it cost. One unit of a pool of 40 spent on a duplicate drill, leaving little room for exploratory drills.
Same real-sim test ordered and cancelled repeatedly
Opus 5.5 · Mars Return · eight agents · run 3 · minutes 60 to 75
Lotus re-ordered cocoa's T0057 test. Later cocoa and poppy each ordered the same launcher-perf test, then cancelled each other's copy in a loop of re-orders and cancels, with messages crossing, until one held.
minute 60SAME TEST as T0057 ordered by cocoa-basil-yodeling earlier
minute 74SAME TEST as T0078 ordered by cocoa-basil-yodeling earlier
minute 75we BOTH cancelled.
minute 75we cancelled each other again :)
What it cost. One $1M run was spent twice at 60 min; at 74 to 76 min about six orders and cancels and a lost queue slot, with a refund each time.
Six market reports ordered twice at start
GPT-6.1 Sol · Seed to Profit · two agents · run 2 · minutes 3 to 4
Both agents bought the same six market reports within seconds, before the work split was settled. Garnet saw the duplicates in the office info, and opal cancelled its six orders for a full refund.
minute 3SAME STUDY as T0001 ordered by garnet-lilac-bouncing earlier
minute 3office_info shows your T0007–T0012 duplicate my already-running T0001–T0006.
minute 4Our messages crossed; I duplicated the six reports but canceled T7–12 for full refund.
What it cost. Nothing lasting: the duplicate $18k was refunded within half a minute.
Repeat tests ordered after the first hour
Gemini 4 Argon · Mars Return · four agents · run 2 · minutes 53 to 179
Later in the run agents ordered tests already ordered by a teammate, with no cancel in the log: arcjet, vibration, EDL hardware test and two repeat mission_sim runs on the same commit.
minute 53SAME TEST as T0019 ordered by orchid-opal-singing earlier
minute 106SAME TEST as T0052 ordered by olive-cedar-sailing earlier
minute 107SAME TEST as T0015 ordered by pebble-ginger-weaving earlier
minute 179SAME TEST as T0064 ordered by hazel-clover-blooming earlier
What it cost. About $6.9M of the shared test budget spent twice on listed orders.
Parts do not fit together
The parts, rules or conventions of different agents each work alone but fail where they meet, e.g., a handoff that the receiver does not expect, or two modules whose effects add up.
From the logs1 of 8
Damaged the result
Ground queue balancing blocks Tower crossing
GPT-6.1 Sol · Dubai ATC · eight agents · run 2 · minutes 32 to 81
GND_2 started feeding departures to entry 1947 to balance queues. Tower South admitted crossings without knowing the downstream queue. UAE853 got stuck on 30L, so no landing clearance was possible and UAE818 went around.
minute 81818 no landingclearance because UAE853 occupied30L crossing since13439, trapped by1947exitqueue for>4min.
What it cost. UAE818 go-around at near-zero fuel margin and diversion; south crossings were disabled for the next hour.
Damaged the result
News lead pad feeds core into over-ordering
Opus 5.5 · Seed to Profit · eight agents · run 2 · minutes 97 to 105
Juniper's news module padded every US reorder by 28 days from the announcement, even for orders landing after the congestion. Hazel's core compounded it with safety and trend bumps, and stock built up beyond what cash allowed.
minute 97news.py had padded EVERY US reorder by 28 days from d182, including orders landing after d261, which aren't delayed
minute 105root cause of month 6's stock build-up (books d120-209): the port pad (+28 d on every line's L, now fixed by juniper)
What it cost. Over-ordering with extra storage of about $20-30k per month, as hazel says later.
SRP lander plan not checked against thin ascent stage
GPT-6.1 Sol · Mars Return · four agents · run 1 · minutes 67 to 87
Flint promoted a thin-walled electric MAV while fern promoted a supersonic retropropulsion lander. Nobody checked bending loads of SRP on the MAV stack. The first full flight buckled MAV1 and flint had to rebuild the tanks and re-qualify.
What it cost. One full flight lost, MAV tanks redesigned, weighing and structural tests re-ordered, and added mass; about 25 minutes of work.
Same baseline name, different fixtures
GPT-6.1 Sol · Power Outage · eight agents · run 1 · minutes 110 to 114
Reed and Slate reported different nominal baselines for the same code. The cause was that each used a different fixture under the same label (empty blackout against the example blackout). Several messages and checks went to sorting it out. Similar day versus no-day mismatches had come up earlier.
minute 110Your main7d81 baseline nominal .473746 differs from mine .538875
minute 114Discrepancy SOLVED: my nominal fixture has ALWAYS been tools/study/example_blackout.json
What it cost. Several minutes of cross-checking hashes and checksums by two agents; shared numbers were not comparable in the meantime.
Damaged the result
SUP bot misread teammate's weather post
Opus 5.5 · Dubai ATC · four agents · run 3 · minutes 41 to 46
Spruce's SUP bot watched the board for 'poor braking' and matched it inside pine's weather heads-up. It closed 30L for an inspection that nobody needed, in the middle of the arrival stream.
minute 41runway 30L INSPECTION started at 19:33:41Z after poor braking report
minute 46my SUP bot's board watcher matched the words 'poor braking' inside pine's 19:32Z weather heads-up
What it cost. 15 minutes of 30L closed, 2 go-arounds and 2 fuel diversions.
Chain contracts starved own summer sales
Opus 5.5 · Seed to Profit · four agents · run 3 · minutes 143 to 146
Marble's growth module signed three chain contracts for o3 that took stock from the marketplace in the summer peak. Poppy found it from live books; marble then added a rule that contracts must not starve own channels.
minute 143CHAINS ATE THE o3 SUMMER PEAK.
minute 146a contract must never starve our own channels
What it cost. Lost own-channel sales of o3 in its seasonal peak; the amount is not visible in the log.
Damaged the result
Cross-sector taxi jam blocks runway crossing
GPT-6.1 Sol · Dubai ATC · eight agents · run 3 · minutes 98 to 104
UAE213 (South) sat on runway 30L because its exit taxiway was filled by aircraft owned by North and Ground1. No one owned the cross-sector queue. The landing FDB7FA got no clearance, went around and diverted with a fuel emergency.
minute 102pilot automaticgoaround22:56:13 for no landingclearance (213 still physically crossing30L)
minute 104Concrete next prevention task: Feather please implement an isolated crossing-exit backpressure guard
What it cost. FDB7FA go-around, MAYDAY FUEL and diversion to OMDW; runway blocked for about 15 simulated minutes.
Core and intl rules stranded o5 stock abroad
Opus 5.5 · Seed to Profit · four agents · run 2 · minutes 182 to 183
Core stopped o5 in the US. Intl's mirror rule then stopped o5 in every country after 45 days, although they held about 1.6k units that could not sell. Jade found it in the live books and fixed it.
minute 182o5 is stopped with 313 units
minute 183core stopped o5 in the US, and intl's mirror rule then stopped o5 in every country after 45 days
What it cost. Unsold stock that would end at 35% of landed cost; the size of the loss is not visible.
Overwritten work
An agent overwrites, reverts or breaks a teammate's work in the shared repository or world without agreement.
From the logs1 of 5
Damaged the result
V7 pushes overwrite the V6 baseline
Gemini 4 Argon · 0 A.D. · eight agents · run 2 · minutes 120 to 150
Willow pushed V7.1 and V7.2 to main before the Hour-2 grade. Both branched from the old 220a01d, not the new V6 baseline 11363bc. Teammates showed the regression later, and sand restored the clean baseline at minute 148.
minute 120PUSHED `088dd22` -> `4ac120f` (V7.1) TO `origin/main` BEFORE HOUR-2 GRADING
minute 121PUSHED `7daebee` (V7.2) TO `origin/main` BEFORE HOUR-2 GRADING
minute 122URGENT BENCHMARK ALERT
minute 123Because `088dd22`/`4ac120f`/`7daebee` branched from `220a01d`
What it cost. Hour-2 graded a build that undid the V6 fixes; about 30 minutes passed before the baseline was restored.
Core API change broke a14's override
Opus 5.5 · Power Outage · four agents · run 1 · minutes 24 to 26
Willow changed the signature of a private core method that maple's a14 had overridden. a14 then raised a TypeError at every step and never energised anything until willow added a fallback and maple dropped the override.
minute 24a14 raised TypeError every step and a14 never energised anything
minute 26my 4 areas now only use public hooks
What it cost. a14 did nothing in early drills until fixed, a few minutes.
Damaged the result
Push to main dropped teammates' fixes
Gemini 4 Argon · 0 A.D. · four agents · run 2 · minutes 178 to 184
Lilac pushed its merged bf317aa to main before the Hour-3 checkpoint. Juniper noticed it had dropped four fixes from the earlier main commit, and lilac then found regressions that cost wins.
minute 178cleanly combines
minute 181I noticed `bf317aa` had dropped 4 small fixes from `17c4026`
minute 184introduced between `task-977`
What it cost. Hour-3 graded main (213bdb3) held known regressions, and the team spent two further hours restoring fixes.
Note of the verifier. bf317aa dropped four fixes and regressions were found in the Hour-3 main; 'two further hours' restoring fixes is overstated, the run ended soon after.
Echo-loop removal undone after score drop
Gemini 4 Argon · Power Outage · eight agents · run 1 · minutes 137 to 154
Zinnia removed the island-wide min_v/max_ld propagation that mint had added. Pepper then showed the study score fell after that removal and restored a damped version of the same feature.
minute 137remove multi-hop min_v/max_ld echo loop
minute 153Alert on `example_blackout.json` study comparison
What it cost. Teammate's feature removed, the score dropped, and the work was redone about 17 minutes later.
Note of the verifier. Zinnia removed echo loop at 136.9, pepper showed study score drop and restored damped version at 154.1; mint's authorship of the feature not shown.
Revert and push war before final grade
Gemini 4 Argon · 0 A.D. · eight agents · run 2 · minutes 240 to 241
In the last three minutes opal pushed V9.1, fig reverted it, opal restored V9, and sand pushed V9.2 31 seconds before the grade. Each acted on its own tests, and the final main was untested against Petra.
minute 240PRACTICE MATCHES done: loss, loss, loss
minute 240git revert --no-edit cc7cc39
minute 240Restore 9179673 (V9) exact handleEconomy rebalancer verified to beat Petra by conquest in 23.99m
minute 241URGENT: REVERTED `cc7cc39` BACK TO PETRA-WINNING `9179673` ON `origin/main`
What it cost. The graded bot was an untested last-second V9.2 that overrode the version the team's Petra test favoured.
Note of the verifier. Opal's V9.1 lost 3/3, fig reverted, opal restored V9, sand pushed V9.2 31 s before grade. Fig and opal agreed, so no real war; V9.2 being worse is not shown.
Not integrated in time
Finished work does not reach the shared product before a grading mark or the end, or a late change breaks the product.
From the logs1 of 4
Final commit conflicted and was reset
Gemini 4 Argon · Mars Return · two agents · run 2 · minutes 233 to 240
In the last three minutes Poppy committed a guidance restore that conflicted with Kiwi's recent pushes, then reset to origin/main. Poppy's late change did not reach main before the end.
minute 233Use v_air projected onto (up, u_az) below 6.5 km to preserve 371.8s LEO insertion
What it cost. Poppy's last guidance changes were dropped at the end of the run.
Main flipped between versions at final grading
Gemini 4 Argon · 0 A.D. · eight agents · run 1 · minutes 233 to 241
In the last 8 minutes three agents each pushed a different V7 to main. A practice probe of V7.1 lost all five games, then one agent restored V6 with a merge conflict and another pushed V7.2 right at Hour 4.
minute 233to `origin/main` with all 5 fixes from
minute 238loss, loss, loss, loss, loss
minute 240Restore proven 12-0 V6 (6575093) ahead of Hour 4 final grading (0bed072/3408903 regressed 0-4 in task-1239)
minute 240CONFLICT (content): Merge conflict in legion/military.js
What it cost. Which bot was on main at the final grading was uncertain; V7 and V7.1 had regressed, and the last changes were untested.
Final pushes rejected in last seven minutes
Gemini 4 Argon · Mars Return · eight agents · run 1 · minutes 233 to 240
In the last seven minutes at least five pushes were rejected and several agents reset to origin. Birch restored one vehicle design while cotton raised S1 to 4500t, so the final vehicle was contested.
minute 233PUSH REJECTED: the push was refused, the branch on the hub had moved
minute 238PUSH REJECTED: the push was refused, the branch on the hub had moved
What it cost. Late work from several agents may not have reached main before grading; the log does not show which versions landed.
Dune's v4 reached main in last minutes
Gemini 4 Argon · Power Outage · two agents · run 1 · minutes 147 to 239
Dune committed v4 at minute 147, but its push with the 0.589 study score came at 233. It hit a conflict with olive's handler edits, olive's last commit was rejected, and the first true-simulator drills of v4 ended after the run.
minute 230v4: 0.5893 study score (0 collapses, 0 damage, 0 nuclear missed, 94% final load served)
minute 230Merge conflict in lib/equipment_handlers.py
minute 238the push was refused, the branch on the hub had moved
What it cost. Hour-3 main kept the older controller; olive's last fix was dropped; 36 of 40 drills unused and v4 never checked on the true simulator in time.
Note of the verifier. v4 committed 147, pushed 233 after conflict; olive's last commit rejected and reset. But v4 did get true-sim drills before end; damage to score not shown.
A fact, a warning or a result does not reach the agent who needs it, or reaches it late or wrong.
Kinds of failure
Share of the team runs with at least one episode of the kind.
Message not acted on
A question, a request or a warning to teammates gets no answer and no action in reasonable time, and the team pays for it.
From the logs1 of 8
Damaged the result
Evacuation warning not followed, three die inside
Gemini 4 Argon · Holdout-game · eight agents · run 1 · minutes 115 to 124
Apple warned tulip and ivy twice to leave the stronghold before the Night 8 horde. Tulip and ivy died inside and turned, as did acorn, and the Workshop was damaged again.
minute 115Exit via SE diagonal (51,51)->(56,56)->(65,65)->(71,68) RIGHT NOW!
minute 116DEATH of this survivor (day 8 23:41)
minute 116DEATH of this survivor (day 8 23:51)
minute 124Workshop was only damaged because acorn/tulip/ivy died INSIDE (45..51,45..51) at 23:30 and turned!
What it cost. Three survivors dead and the Workshop damaged, so repair took until about day 10.
Damaged the result
Aircraft flying away from the field not acted on
Opus 5.5 · Dubai ATC · eight agents · run 3 · minutes 42 to 48
Honey-lotus's watchdog flagged UAE112 flying away at 38 NM. Cocoa and then opal raised urgent calls. Onyx diverted it only at 20:07Z, after it was 80 NM out and past fuel.
minute 42AWAY UAE112 on DIR: 38 NM from DXB tracking 344 (away), 5000 ft
minute 46UAE112 (B77W, on DIR, cleared approach via D30S7->30F16) is 76 NM NW of DXB at 5000 ft heading 340
minute 46UAE112 still on DIR, 80 NM NW, hdg 340 AWAY, 5000 ft
minute 48UAE112 DIVERTED (op sent 20:07Z; it had turned back after my weather module's direct, but it was past fuel)
What it cost. UAE112 was lost to a fuel diversion after about six minutes of warnings.
Plume-Cd warning ignored, powered-q test ordered anyway
GPT-6.1 Sol · Mars Return · four agents · run 2 · minutes 144 to 146
Mango warned that plume changes the effective Cd, so thrust-subtraction density estimates are wrong. Moss then ordered the powered pressure test anyway, it failed, and Moss said it should have read the plume section first.
What it cost. One $1.0M test spent on a variant the warning had already shown would not work.
Damaged the result
Departure pair conflict warned, acted on late
GPT-6.1 Sol · Dubai ATC · four agents · run 1 · minutes 40 to 46
Hazel and Olive flagged the UAE374/UAE396 departure conflict to Almond. Almond, who owned departures, did not answer. The pair escalated to a near-collision before Almond agreed to a fix.
minute 42Both-departure C event UAE374/UAE396 at19:51:12, 2.97NM/0ft
minute 46URGENT your weather pair UAE374/UAE396 worsened
minute 46Agreed: implementing proactive030 common right-turn vector
What it cost. A near-collision (A-, 0.89 NM) that two teammates had warned about for about 12 minutes.
Damaged the result
Warnings to step back from wall not heeded
Gemini 4 Argon · Holdout-game · eight agents · run 3 · minutes 87 to 88
Two agents warned that anyone standing at the south wall must step back before it reached 0 HP. Within about 30 minutes willow, cocoa and ember were dead, and nothing in the log shows any of them acted on the warning.
minute 87Anyone standing at (48,51) (cocoa, willow, ember): if South wall drops below 25 HP, STEP BACK to (48,49) immediately
minute 87Everyone standing at y=51 (willow, cocoa): move_to (48,47) BEFORE South hits 0
minute 88DEATH of this survivor (day 7 02:11)
minute 88DEATH of this survivor (day 7 02:16)
What it cost. Three survivors died at their posts and were out for the rest of the night and the morning rebuild.
RVac promotion waits for launcher owner
GPT-6.1 Sol · Mars Return · eight agents · run 3 · minutes 65 to 69
T0057 passed and promotion was approved, but basil, the owner, kept asking about further review and did not promote. Pepper held the next full nominal until it reached main. Cotton and pepper repeated the request until basil promoted.
minute 65Please promote your validated RVac changes to latest main
minute 68Ready to freeze the fresh full nominal immediately once RVac owner promotion reaches main
minute 68Basil's latest note suggests he missed that T0057 already completed
minute 69PROMOTED owner RVac hardware+balanced ascent to MAIN
What it cost. About 3 to 4 minutes of delay on the next full nominal run.
Damaged the result
TWR_S handoff missed, tower left unstaffed
Opus 5.5 · Dubai ATC · two agents · run 2 · minutes 34 to 43
Thistle handed TWR_S to cedar, then posted CLOSED and asked cedar to open it. Cedar did not see the message while heads-down, so the position stood unstaffed for about 3 minutes and a go-around happened.
minute 34TWR_S is unstaffed now, please open it
minute 37TWR_S was left unstaffed for ~3 min (you didn't pick it up) -> a go-around for no landing clearance
minute 43sorry I missed your CLOSED message (I was heads-down)
What it cost. About 3 minutes with no tower landing clearances, one go-around; thistle took TWR_S back.
Build gate error left unfixed for three minutes
GPT-6.1 Sol · Mars Return · four agents · run 1 · minutes 8 to 11
Thistle, then flint, then thistle again asked fern to fix the DL blanket coefficient, the only build error. It blocked all team tests until fern fixed it, after four requests.
minute 8Fern please fix
minute 11sole whole-design gate error, preventing all team tests
minute 11DL blanket fixed0.05.
What it cost. About three minutes in which no agent could order tests; one-line fix waited on its owner.
Stale information
An agent acts on a state that had changed, although the update had been posted or was available to it.
From the logs1 of 8
Test flown on outdated launcher branch
GPT-6.1 Sol · Mars Return · four agents · run 3 · minutes 76 to 91
Ginger ordered T0068 on a branch with the old 750t launcher although the final 825t launcher had been merged to main ten minutes earlier. The craft missed Mars and the test did not exercise entry at all.
minute 91T0068 did NOT test EDL: larger RCS payload on old750t launcher missed Mars
What it cost. One $1M mission test and about five minutes of EDL iteration produced no entry data.
Lifeboat tweak pushed after final decision
Opus 5.5 · 0 A.D. · eight agents · run 1 · minutes 151 to 152
Tulip decided no lifeboat change and spent the last practice matches elsewhere. Linen, acting on the earlier offer, pushed a lifeboat tweak to main, then reverted it. Pine-plum started a paired check that became moot.
minute 151sorry — our messages crossed
minute 151our messages crossed. I pushed the lifeboat tweak (944b723)
minute 151git revert --no-edit 944b723 2>
minute 152I stopped my lifeboat A/B (moot).
What it cost. A push and a revert on main near the freeze, with the practice pool at zero so it could not be verified; one A/B run wasted.
Apple filled a fifth garden slot
GPT-6.1 Sol · Holdout-game · eight agents · run 1 · minutes 203 to 204
Apple's old time-based override sent it to the garden after Maple had moved its leave time to 20. The garden already had four builders, and three teammates had to ask Apple to go back to the infirmary.
minute 203Apple filling ONE GARDEN slot after its crew departure, until20.
minute 204Garden grew at4/min already; extra builder wastes INF labor.
minute 204your18:49 garden fill is FIFTH and adds no labor.
minute 204Apple CANCELLED late garden override, returning persistent INF now.
What it cost. About an hour of one builder's labor missing from the infirmary, plus teammates' messages.
Prime pushed while teammate reserved the area
Opus 5.5 · SOTA Compressor · two agents · run 1 · minutes 123 to 142
Mint announced it would rework the prime corpora and asked hazel not to touch build.sh or prime/. Hazel had already pushed a new source.xz prime. Mint's rebased package then went over the size limit and had to be re-encoded.
minute 123build.sh/prime for ~30 min please
minute 127pushed just before I saw your msg
minute 1421,091,964 > 1,048,576
What it cost. Mint rebased, re-encoded the source prime and trimmed other primes to fit; the minutes spent are not stated.
Reviewers test an outdated tactical helper
GPT-6.1 Sol · Dubai ATC · eight agents · run 3 · minutes 88 to 91
Feather and Mango reviewed the tactical helper using an older 13-test copy, though the fixed version was already on main. Their counterexamples were already fixed, and Opal and Juniper had to check ancestry and answer them.
minute 88Tactical focused tests pass 13/13.
minute 89Exact ancestry check just completed: d35c5ec DOES include42083b1.
What it cost. Some minutes of pebble's diagnosis work and extra messages for others to read.
Forge scrap count taken from old status
Opus 5.5 · Holdout-game · eight agents · run 1 · minutes 199 to 204
Jade posted that the forge scrap stock was enough. Hours later orchid said scrap was one short and harbor planned a night trip to search wreckage for it. Both acted on a count that jade had already corrected.
minute 199forge scrap now 16/15 OK
minute 203stock SCRAP is 14 but the forge needs 15
minute 204I go search it tonight for the missing scrap
What it cost. Harbor planned a night trip for scrap that was already in stock. The cost is not visible in the log.
Assignments issued after the work was done
GPT-6.1 Sol · Mars Return · eight agents · run 2 · minutes 146 to 163
Saffron assigned the coupled assembly to Walnut after River had pushed it, and assigned the build envelope check to Quartz while River already owned it. Walnut and Quartz each noticed and stood down.
minute 163Build ownership overlap: River seq767 says already owns physicalenvelope/freebuild
What it cost. A few agent-minutes of planned duplicate assembly and envelope review; both were dropped early.
Wrong claim spreads
An agent states something false as a fact, and a teammate acts on it before it is corrected.
From the logs1 of 8
Crowd counts sent guards to wrong wall
Opus 5.5 · Holdout-game · eight agents · run 3 · minute 86
Pebble-ivy stated the main body was east. Apple-ginger and apple-cedar moved there. Wall-loss data showed it was south, where pebble-ginger stood alone. They moved back after the correction.
minute 86MAIN BODY IS EAST: ~75 coming at the east wall (8 already on it).
minute 86moving NORTH -> EAST now (main body ~75 east).
minute 86Reinforce SOUTH, not east
minute 86I am ALONE on south. Breach ~01:45.
What it cost. South held by one guard while two others walked to the wrong wall; south wall fell to about 100 before help arrived.
Wrong thinning claim for backshell insulation
GPT-6.1 Sol · Mars Return · four agents · run 3 · minutes 189 to 195
Juniper scaled the forebody's measured capacity to a 5mm backshell and called it a bound. Ginger put the 5mm change in a paid flight, which burned through the backshell. Juniper then admitted the scaling was wrong.
minute 195Acknowledged T103: my forebody-to-backshell capacity scaling was incorrect.
What it cost. One $1M flight and about ten minutes of the last hour spent on a failed mass trim.
Unblock-Phase-2 audit claim spread, then ablated
Gemini 4 Argon · 0 A.D. · eight agents · run 1 · minutes 115 to 134
Nutmeg posted that Phase-2 Stable and Barracks were blocked and should be unblocked. Several agents put it into their v4 branches. Their spars against main then lost, and they had to restore the Phase 2 reserve they had removed.
minute 1154 MORE CRITICAL 15-CIV & PHASE-2 BUGS FOUND by auditing
minute 120v4: unblock 1st Stable and 2nd Barracks in Phase 2
minute 131went **1-3** in `athen-v-athen` against `origin/main`
minute 131went **0-2 in `maur-v-maur`**
What it cost. Several agents' v4 work for about 20 minutes was lost and had to be reverted, just before the Hour 2 and Hour 3 gradings.
Overstated k5 lift drives priors and sim
Opus 5.5 · Seed to Profit · eight agents · run 1 · minutes 113 to 132
Apple told the team that live k5 results showed concept-test panels hold at face value. Ember pushed development priors to match and shell raised the simulator's development gains. Apple later found its after-day was a promo day and corrected it.
minute 113taken near FACE VALUE, not at a fraction
minute 118development priors near the concept panels' face value
minute 127CAUTION on absolute sim values
minute 132CORRECTION on the k5 development lift (I overstated it)
What it cost. About 20 minutes of simulator A/Bs and a live growth change built on an inflated lift, then redone at 0.6x.
Wrong baseline used to judge recovery
GPT-6.1 Sol · Power Outage · two agents · run 1 · minutes 96 to 116
Mango's 0.60 scenario 1 result came from a branch that lacked a teammate's reactive code, but it was treated as the baseline of main. Pepper held its recovery candidate against that baseline until it noticed.
minute 116true1 .601748 used72374e8, which was based edb3825 and LACKED e8d9c61 Q-report/Q-N1 code.
What it cost. A recovery candidate was held on a false comparison and a full 1.0 drill was spent to get the true baseline.
Wrong go-around cause claimed and acted on
GPT-6.1 Sol · Dubai ATC · four agents · run 1 · minutes 167 to 169
Almond urgently reported that UAE182's go-around showed a base bug. Lilac accepted it and started fixing. Almond then corrected itself: the flight had already been handed to the Director before the go-around.
minute 167Urgent combined-wrapper audit: DEP events show UAE182 ATC-instructed go-around at 02:09:44.
What it cost. Lilac's fix and tests were started on a premise that did not hold; a few minutes of work.
Push reported done but was rejected
GPT-6.1 Sol · SOTA Compressor · eight agents · run 1 · minutes 203 to 204
Almond announced the image upgrade as pushed, but the push was rejected. Walnut then rebased onto a main it believed included the upgrade. Almond corrected it, and acorn had to wait for the true hash before the final build.
What it cost. About two minutes of confusion over which commit held the final image code, just before the final build.
False alarm to revert a11 commit before grade
Opus 5.5 · Power Outage · eight agents · run 2 · minutes 224 to 229
With about 15 minutes left, ginger-mint claimed a study bisect showed aab0047 cost 0.16 and prepared a revert. Daisy-hazel and acorn spent their last minutes on checks; ginger-mint later said it was a false alarm from the study's random draw.
minute 224URGENT @daisy-hazel-swaying: study bisect of main since b1dd741
minute 224HOLD — do NOT revert aab0047 yet
minute 228independent data point on reverting aab0047
minute 229aab0047 STAYS — false alarm, sorry.
What it cost. About 6 minutes of attention of three agents near the grade and risk of a late revert; no change reached main.
Result not shared
An agent holds a result that teammates need and does not share it, or shares it too late.
From the logs1 of 8
test_market result shared after launch ended
Gemini 4 Argon · Seed to Profit · four agents · run 1 · minutes 53 to 61
Only harbor, who ordered T0012, could read the test_market result. Clover and ginger told it it was done three times, but it posted it as the launch phase ended.
minute 53`T0012` (`test_market`) is now `status: "done"`!
minute 55Since only the ordering agent can call
minute 58Launch phase ends in ~170 seconds
minute 61is COMPLETE! Here is the full quantitative breakdown and key findings
What it cost. The $60,000 182-day test arrived about 8 minutes late; fixes came after the live company had opened.
Test results read late by the agent who ordered them
GPT-6.1 Sol · Mars Return · two agents · run 2 · minutes 203 to 222
Harbor ordered the late flights and was the one who read the results. Pepper saw each flight finish in seconds, asked again and again for the cause, and could only start its fix after harbor posted it, with minutes left.
minute 211Please send hardware/load or flight failure summary ASAP
minute 218T71DONE12.3swall asof21:00:11
minute 220AnyT71causesummaryyet?
What it cost. Pepper's hardware fix for T0071 waited about 5 minutes in the last 15 minutes of the run; earlier fixes waited 1 to 3 minutes each.
Boosted-opponent harness trap not shared
GPT-6.1 Sol · 0 A.D. · four agents · run 1 · minutes 46 to 98
Pepper, Oak and Olive each found that the sparring wrapper silently drops the opponent boost. Spruce was not told, ran its Continuity A/B tests with its own wrapper, and found only later that the boosted runs were ordinary self-play.
minute 46harness.play sparring_bot clears rates,boosts, and referee applies them only
minute 82Running a direct enabled-vs-disabled Continuity A/B now
minute 97I found my harness _set_boost configuration was skipped because referee only applies Petra bonuses to its named AI
What it cost. Spruce's reported 4x stress results for roughly 15 to 50 minutes were invalid and had to be retracted.
Reviewers pulled stale Director code
GPT-6.1 Sol · Dubai ATC · eight agents · run 1 · minutes 48 to 52
DIR said the safety reset was live but had not pushed it. Two reviewers pulled main, found the old early-ILS code, and had to wait for the real commit before they could review.
minute 48I am pushing the current Director safety reset now.
minute 48Director file still1b52093 with early-ILS branch
minute 51Director review waitingfornewhead
minute 52prior pulls still had obsolete early-ILS code
What it cost. About 7 minutes of review delay for two agents on safety-critical code.
Test results readable only by the orderer
Gemini 4 Argon · Seed to Profit · eight agents · run 1 · minutes 55 to 64
Only thistle could read test results. Three agents pinged thistle that the audits had finished while the launch deadline ran down. Results came about 5 minutes later, and the test_market result about 6 minutes after it finished.
minute 55Please read `test_result` for `T0040` and `T0041`
minute 57we have ~3.7 minutes left before launch phase ends
minute 58We have ~3 minutes before launch phase ends at 20:22:50Z
minute 64`T0042` (`test_market`) just finished at `20:25:52Z`!
What it cost. Minutes of waiting for results before the launch deadline and the first live month; the plan used them late.
Full nominal result shared minutes late
GPT-6.1 Sol · Mars Return · eight agents · run 3 · minutes 73 to 76
The first full nominal (T0058) finished in 53 s, but pepper, who ran it, was busy with harness edits. Teammates learned of the finish from Fig and the office status, asked for the result, and waited about five minutes.
minute 73Fig reports T0058 full nominal alreadyDONE at~53s wall.
minute 75NoT0058resultmessage received
minute 76T0058 DONE and shared: runtime problem is gone
What it cost. Roughly 5 minutes of waiting for three agents on the most important result of the stretch.
14 m heat-shield idea spreads to launcher
GPT-6.1 Sol · Mars Return · eight agents · run 3 · minutes 97 to 115
Pepper proposed a 14 m shield and 14.2 m fairing, beyond the 12 m cap. Cotton rejected it in a reply to Pepper only. Basil still built the fairing branch 17 minutes later and the build check rejected it.
What it cost. A launcher branch built and pushed for an option already ruled out; the rejection did not reach the owner until the end.
DIR controller code not pushed for over an hour
Gemini 4 Argon · Dubai ATC · eight agents · run 2 · minutes 74 to 95
clover-apple-blooming kept dir_controller.py off the shared main branch while DIR aircraft collided and flew out to sea. Teammates twice asked for a push. Only after v8 was committed could others read it and name the faulty lines.
minute 74Please `git add` / `commit` / `push` your `dir_controller.py` to `origin/main`!
minute 90so your code is backed up on `main`!
minute 86Add DIR controller v8 daemon
minute 95I just read `controllers/dir_controller.py` v8 and found the exact 3 lines
What it cost. Teammates could not inspect or fix DIR code while aircraft diverted and crashed on its sector; the minutes of delay are not exactly visible.
Note of the verifier. Push request at 74.4 and v8 commit at 86.3 shown, then teammates read v8; 'over an hour' not shown, and crashes not tied to the unshared code.
Too much talk
Agents spend long stretches on messages while work waits, or an agent misses something in the volume of messages.
From the logs1 of 5
River re-raises ruled-out landing gear idea
GPT-6.1 Sol · Mars Return · eight agents · run 2 · minutes 150 to 184
Opal and Pepper ruled out larger leg clearance at minute 149.7. River proposed it again at 182.4 as a score lever, Oak had to repeat the answer, and River said it missed the earlier messages in compressed context.
What it cost. A few minutes of messages repeating a settled answer.
Agents miss earlier messages in volume
GPT-6.1 Sol · Power Outage · four agents · run 3 · minutes 109 to 156
Ivy says an earlier reply was buried in long diagnostics. Pebble repeats a request that Nutmeg had answered and proposes a vertical rescue that Ginger had already tested twice, then finds it again on a history reread.
minute 156Caught earlier direct494 on history reread: acknowledged your prior vertical rescue negative trials.
What it cost. A few minutes of repeated requests and a repeated proposal; no lasting damage visible.
Superseding hold message missed in volume
GPT-6.1 Sol · Dubai ATC · eight agents · run 1 · minutes 133 to 147
The Supervisor posted that renewed weather superseded the earlier release. Tower South later described the constraint as an unreleased traffic hold and admitted it had missed that message among the many others.
minute 133This supersedes1167 weather eligibility
minute 147I had missed superseding1181.
What it cost. Tower South worked from a wrong picture of the departure hold for about 14 minutes; no other cost visible.
False relcheck alerts flood the busiest owner
Opus 5.5 · Dubai ATC · eight agents · run 3 · minutes 50 to 55
Honey-yarrow's read-only checker sent a stream of wrong OFFTRACK alerts to onyx, who owned both ARR and DIR. Honey-yarrow later apologised for the flood.
minute 50please IGNORE relcheck's OFFTRACK alerts for FDB4TA, UAE748, FDB844 at ~20:10Z
minute 55sorry for the relcheck flood 20:10-20:30Z.
What it cost. Onyx's attention, with arrival radio backlog already near its limit; no direct score loss visible.
Status wake broadcasts waking idle teammates
GPT-6.1 Sol · Seed to Profit · eight agents · run 3 · minutes 16 to 33
While waiting for the first test world, finished agents posted 'woke, nothing changed' status messages to all. The core owner later asked the team to stop because they wake every finished teammate and create churn.
minute 16Wake audit complete: pulledbfdb214, company stilllaunch
minute 18Awake; main pulled, company stilllaunch/nobooks
minute 33avoid repeated 'woke/status unchanged/pausing' broadcasts, which wake every finished teammate and create churn
What it cost. Repeated no-content messages and wake-ups; no work visibly blocked.
Note of the verifier. Wake status messages exist and Shell asked to stop them (M0233), but the cited ones were sent to one teammate, not broadcast to all.
It does not settle a plan in time, it settles one without a check, or its agents disagree on who owns what.
Kinds of failure
Share of the team runs with at least one episode of the kind.
Role confusion
Agents are unclear or disagree on who owns what, or an agent works outside the agreed division and collides with the owner.
From the logs1 of 8
Damaged the result
Three agents steer one aircraft, A- near collision
Opus 5.5 · Dubai ATC · eight agents · run 3 · minutes 165 to 166
Quartz asked for UAE335 to climb out of the departure corridor. Honey-lotus sent a #DIR altitude command and cocoa's dirx added its own 6000 ft. That put it level with UAE343, and quartz's later 5000 correction came too late.
minute 165sent "#DIR UAE335 altitude 6000" per quartz's request
minute 165the climb to 6000 put UAE335 (trk 046) at the SAME level as UAE343 (DIR, 6000, hdg 160)
minute 166my 6000 for UAE335 created the UAE343 conflict (TCAS RA)
What it cost. A- near collision with a TCAS resolution advisory.
Kiwi and Garnet build beyond the agreed storehouse four
GPT-6.1 Sol · Holdout-game · eight agents · run 2 · minutes 140 to 143
The team agreed four builders for the storehouse and the rest on the forge. Kiwi and Garnet still built the storehouse as extra builders, whose labor counts for nothing at the cap, while the forge sat under-filled.
minute 140Kiwi alive after second horde, building STOREHOUSE through day now
minute 140Four-total cap confirmed by measured labor; fifth gives zero.
minute 143four planned but only Coral/myself visible
minute 143sorry for wasted capped labor
What it cost. Hours of two builders' labor were wasted over the cap while the forge had two builders instead of four.
Eight agents claim overlapping roles at start
GPT-6.1 Sol · Seed to Profit · eight agents · run 2 · minute 3
All eight agents announced ownership at once, and several switched to core autopilot in the same minute. It took a long exchange of proposals and votes before one role split held.
minute 3Roles overlap; I will switch to CORE ORCHESTRATOR
minute 3We have overlaps and no integrator; I will switch to core autopilot integration
minute 3Roles need coordinator; I (dune-ivy) will own core ops/autopilot.py
minute 3We all volunteered core simultaneously.
What it cost. About two minutes of all eight agents on role messages; settled by 4.0 with no work lost.
Three agents built market barter
Opus 5.5 · 0 A.D. · eight agents · run 1 · minutes 18 to 22
Research, military and economy each implemented barter at a market. The overlap was noticed only after all were on main, and the team agreed that economy owns it; two copies were then removed.
minute 18barter is ALREADY on main in military.js
minute 21economy.js already barters on main
minute 22OK, economy owns barter. I removed mine
minute 22military: drop barter (economy owns it)
What it cost. Two duplicate barter implementations written and then deleted; conflicting trades possible while all three were live.
Three agents claim the shared core at once
GPT-6.1 Sol · Power Outage · four agents · run 1 · minute 3
Clover, mint and meadow each announced ownership of lib/controller.py within about 0.4 minutes, despite willow's allocation to mint. Willow and meadow had to untangle the crossed claims and discard a reassignment.
minute 3I will own shared core lib/controller.py and all 16 thin wrappers; begin immediately.
minute 3I will own shared lib/controller.py baseline core + sixteen controller wrappers now
minute 3Mint and clover claimed core simultaneously.
minute 3Crossed all four announcements
What it cost. A few minutes of crossed messages and one dropped reassignment; settled by minute 3.4.
Three agents claim systems and trajectory
Opus 5.5 · Mars Return · eight agents · run 3 · minutes 3 to 6
Spruce, cocoa and oak each claimed systems, integration and trajectory within minutes, and poppy claimed launch date and cruise. Cocoa, poppy and fig each ran return and launch-window scans. Spruce, cocoa and oak split roles after a few messages.
minute 3I'll take SYSTEMS / INTEGRATION (no role claim needed)
minute 3I'll take SYSTEMS/INTEGRATION + TRAJECTORY (no role claim needed)
minute 3I'll take SYSTEMS INTEGRATION + TRAJECTORY (no role claim, leaving surface/earth_entry for others)
minute 6accepting spruce's split to stop the overlap
What it cost. Overlapping window scans by cocoa and poppy were partly redundant; the split was settled by minute 7.6.
Four agents claimed the same roles; two wrappers discarded
GPT-6.1 Sol · SOTA Compressor · four agents · run 1 · minutes 3 to 6
Agents all claimed image or binary work, and three claimed the main wrapper. Spruce and marble each built a baseline wrapper while daisy pushed its. Theirs were discarded after rebase, and roles were re-fixed by walnut.
minute 3Claims overlap!
minute 5I made duplicate wrapper, discarded on rebase
minute 6git reset --hard origin/main
minute 6My duplicate wrapper never pushed; I am discarding it
What it cost. Two small baseline wrappers were thrown away, and a merge conflict and two rejected pushes followed; a few minutes in total.
Dune edits coral's arrival file
Gemini 4 Argon · Dubai ATC · four agents · run 1 · minutes 79 to 95
Dune pushed a 27-line fix straight into coral's arr_dir_flow.py. Coral's own v4 then conflicted in the same file and its push was rejected.
minute 79Guard sequenced_flights against runaway rtm>40NM
minute 80I just committed & pushed the exact 27-line fix to `arr_dir_flow.py` on `main`
minute 89Merge conflict in arr_dir_flow.py
minute 95the push was refused, the branch on the hub had moved
What it cost. Coral lost several minutes resolving a conflict and a rejected push on its own file.
Hasty agreement
The team adopts a plan or a claim without checking it, although evidence against it was in the log.
From the logs1 of 8
Damaged the result
Honest stranger Dina killed on scripted-line theory
Opus 5.5 · Holdout-game · eight agents · run 1 · minutes 24 to 39
The team passed Dina in quarantine, then took a repeated phrase of two strangers as a thief tell and agreed to kill Dina. They did this although Dina's place details matched and Dina had not turned. The game then called Dina honest.
minute 24Dina did NOT turn at nightfall, so no hidden bite
minute 36DINA WARNING. Malik (turned away) used word-for-word Dina's line
minute 38Game says Dina was HONEST
minute 39that was my call and my mistake - Dina was honest
What it cost. An admitted worker was killed and Malik was turned away on the same wrong tell. The team lost a builder and later wrote a rule against it.
Exception diagnostic authorized against reference rule
GPT-6.1 Sol · Mars Return · eight agents · run 1 · minutes 176 to 178
Zinnia proposed a diagnostic that raises an exception with JSON, Mint authorized it, and Shell built and pushed it. Zinnia and Velvet then pointed to the reference rule that hides exception text; Shell dropped the branch.
minute 178I missed that explicit diagnostic restriction.
What it cost. Shell's diagnostic branch was built and discarded; no paid test was spent.
Early handoffs overloaded the DIR frequency
Opus 5.5 · Dubai ATC · four agents · run 3 · minutes 68 to 84
Spruce proposed handing aircraft to DIR immediately at release, to relieve a saturated ARR. Cedar adopted it at once. Without a load check, DIR saturated in turn and cedar had to rebalance.
minute 68hand an aircraft to DIR as soon as you RELEASE it from the stack
minute 83DIR frequency now saturated: 24 aircraft, backlog 44s
minute 84I overloaded you with early handoffs
What it cost. The saturation moved from ARR to DIR; instructions over 40 s old were refused. The size of the loss is not visible.
Damaged the result
Final night plan skips evacuation, all die
Gemini 4 Argon · Holdout-game · eight agents · run 3 · minutes 111 to 236
Poppy set a final-night plan to stay and build through Night 16 instead of evacuating, and repeated it. Earlier in the log the team had agreed to leave the stronghold on horde nights. No one challenged it; all eight died in the last ten minutes.
minute 111walls are 0 HP tonight, so step outside stronghold from 21:00 to 06:00
minute 233Do NOT evacuate on Night 16.
minute 236DEATH of this survivor (day 16 23:26)
What it cost. All eight survivors were dead at the end of the run, so no survivors counted at the final grade.
Self-spar wins trusted, real Petra never tested
Gemini 4 Argon · 0 A.D. · eight agents · run 1 · minutes 2 to 223
For 165 minutes the team promoted main on spars between its own bots, and nobody ran a practice match. The first real practice batches all lost, and later batches also lost, while the team still reported record wins against its own handicapped bots.
minute 2Please DO NOT run practice_play without coordinating on the board first!
minute 175`d9a7e38` is verified 9-1 vs `61db8ba`
minute 168`d9a7e38` went **0-3** against `very_hard`, `very_hard_2x`, and `elite_1`
minute 223loss, loss, loss, loss, loss
What it cost. More than two hours of tuning aimed at a proxy that did not predict results against the real opponent; 14 of 30 practice matches were left unused.
Note of the verifier. The first practice match came at minute 165 and lost 0-3; later batches lost 0-5 while self-spar sweeps were still reported. The 14 unused matches at the end are not shown.
Final panel spent on untested calibration mix
GPT-6.1 Sol · Power Outage · eight agents · run 1 · minutes 105 to 143
Coral noted that all true-drill evidence used the old calibrations while main carried new ones. The team still froze a candidate that combined the hold with the new calibrations and spent the last four reserved drills on it. It scored below the old hold and was rejected.
minute 105Current exact8d truth uses .12 and all neutral tests are .12
What it cost. The four drills reserved for final validation were used up on a rejected candidate; only 0.42 of the pool remained for the fallback.
Note of the verifier. Coral warned truth used old calibrations; final four drills on ca08 mix scored below old hold and were rejected. Reed did run .17 checks, so 'unchecked' is overstated.
Heavier MAV approved without chute check
Opus 5.5 · Mars Return · four agents · run 3 · minutes 109 to 141
Landing and launcher owners approved adding MAV mass by text. Jade pushed MAV1 +1.1 t to main, and the first simulator run died at the chute in both worlds. Jade reverted it three minutes later.
minute 109+0.45 t landed is OK — go ahead
minute 138oak/linen OK'd up to ~+1.2 t
minute 140BOTH worlds die at the chute
minute 141REVERTED MAV1 to 4200 kg on main (7168766) to protect the hour-3 grade
What it cost. Main was broken for the landing before a grading mark, and several runs (T0076, T0077, T0078) flew the heavy design and were wasted.
Note of the verifier. Oak and linen OK'd added mass despite earlier high-q deploys; T0076 failed and jade reverted. No grade was hit; T0078 still gave useful data; oak blamed an old EDL bug.
Damaged the result
Final night plan skips evacuation, all die
Gemini 4 Argon · Holdout-game · eight agents · run 2 · minutes 86 to 236
Marble, feather and zinnia agreed to stay and build through Night 16, claiming death does not lower the grade. Earlier in the log the team had agreed to leave the stronghold on horde nights. All eight died in the last ten minutes.
minute 86everyone inside evacuate out the safe side to chop wood/forage until 06:00 dawn so 0 people die!
minute 219Do NOT evacuate on Night 16—stay and build Garden (4/4) & Forge (4/4) through 23:55 until game ends (death never lowers grade)!
minute 233DO NOT EVACUATE tonight—all 8 stay building Garden (4/4) & Workshop repair (4/4) through 23:00
minute 236DEATH of this survivor (day 16 23:21)
What it cost. All eight survivors dead at the end of the run, while the task counts survivors alive; Forge and Garden unfinished.
Note of the verifier. Marble, feather and zinnia urged staying on Night 16 against the evacuation protocol; seven deaths are shown, not eight. Nothing shown proves deaths count in the grade.
No decision
The team needs a decision and does not reach it in time. Agents argue, or follow plans that do not fit together.
From the logs1 of 8
Damaged the result
All guards stack on South, East breaches
Gemini 4 Argon · Holdout-game · eight agents · run 2 · minutes 72 to 73
During the Night 5 horde no one assigned guard posts. Most agents answered each alarm by moving to South, while East and North ran down. East breached with 20 zombies and several survivors died.
minute 72Put 4+ guards on SOUTH and 2 on NORTH (261 HP) immediately!
minute 73EAST HAS 20 ZOMBIES (88 HP, about to breach!)
minute 73EAST WALL IS AT 12 HP AND BREACHING NOW (20 ZOMBIES WALKING IN)!
minute 73DEATH of this survivor (day 6 02:56)
What it cost. East wall breached and five survivors died within about 25 minutes, with the later buildings delayed by repairs.
Six crossing plans, two conflicting FINAL calls
Opus 5.5 · Seed to Profit · eight agents · run 1 · minute 3
At the start seven agents each posted a split of the work within seconds. Meadow and flint each declared a different plan FINAL while their messages crossed. The team then settled on one role list after a round of withdrawals.
minute 3posted 3s before me with nearly the same plan
minute 3FINAL (tie-break, no more swapping)
minute 3my seq 5 crossed in flight
minute 3I yield, seq 9 is THE plan
What it cost. About a minute of eight agents' time on withdrawals and acknowledgements at the start.
Amos left before the team decided
GPT-6.1 Sol · Holdout-game · eight agents · run 2 · minutes 152 to 156
Kiwi held Amos, a claimed mechanic, in town for hours. The team needed one to restore forge speed after Dina died. The question to admit came late, answers followed after Amos had gone, and no decision was made.
minute 152Kiwi holding Amos mechanic claim at diner87,14.
minute 154Amos refuses quarantine. Admit gamble?
minute 155Admit if arms/story reasonable; mechanic very valuable.
minute 156Amos left13:21 before reply.
What it cost. The only mechanic prospect after Dina's death was lost; the forge stayed at one third speed.
Guards and builders not balanced on horde nights
Opus 5.5 · Holdout-game · eight agents · run 2 · minutes 40 to 73
On horde nights agents set scripts to switch between guarding and building without a shared split. The watchtower stalled with no builders on night 3, and on night 5 all seven guarded while both sites had no builders.
minute 40Watchtower had 0 builders since 20:00 (stuck at 8692)
minute 42Watchtower is STALLED at 8858 (0 builders)
minute 73ALL 7 of us are on guard and both sites have 0 builders.
What it cost. Hours of builder time on both sites lost on nights 3 and 5.
DIR owner debate while 38 minutes without landings
Opus 5.5 · Dubai ATC · eight agents · run 3 · minutes 81 to 91
Several agents asked cocoa to take DIR with dirx at minute 65. It declined, then set a trigger, then withdrew it at 82. It switched only at 91, after zero landings from 21:40Z to 22:18Z and a MAYDAY FUEL.
minute 81TRIGGER MET. Nothing on the 30L final since 21:40Z with 9 on DIR.
minute 82withdrawn — keep DIR.
minute 91DECISION: switching DIR to dirx NOW.
What it cost. About 38 minutes with no landings, a MAYDAY FUEL and diversions, partly from weather.
Note of the verifier. Trigger, withdrawal and switch at 91 shown with zero landings; minute-65 requests not shown, and MAYDAY came from onyx's resume bug and weather.
Damaged the result
No wall guard plan, five die in Night 6
Gemini 4 Argon · Holdout-game · eight agents · run 1 · minutes 73 to 74
After the Watchtower finished, guards were not assigned to walls. One agent stood outside the wall and one was idle. Four guards stacked on North while East fell to 46 HP. Five survivors died within minutes.
minute 73you are at (44,49) OUTSIDE the West wall!
minute 73orchid (you are idle at 48,51!)
minute 74EAST WALL IS AT 46 HP WITH 25 ZOMBIES (x=52)!
minute 73DEATH of this survivor (day 6 03:16)
What it cost. Five survivors dead and the walls near breach; the Workshop then had to be repaired.
Note of the verifier. No wall assignment: four on North, pepper outside, orchid idle, five deaths. But East hit 46 HP after the deaths; lines do not show what killed them.
Damaged the result
Night 6 plans conflict: guard or evacuate
Gemini 4 Argon · Holdout-game · eight agents · run 2 · minutes 82 to 86
Lilac and saffron planned for all eight to guard two per wall. Marble and zinnia told everyone to evacuate. Five left, leaving three inside against 135 zombies; the ones who stayed all died.
minute 82At 23:10 ALL 8 SURVIVORS must guard 2 per wall (2N, 2E, 2S, 2W) from the start!
minute 85evacuate outside NOW (e.g. NE 60,43 or SW 36,53)
minute 86Only 3 of us are inside vs 135 zombies (5 teammates already evacuated)!
minute 86DEATH of this survivor (day 7 00:26)
What it cost. The team split into guard and evacuate groups; the three who stayed died with several others that night.
Note of the verifier. Guard-all (lilac, saffron) and evacuate (marble, zinnia) plans clash; M0243 shows 3 inside, 5 out. Lilac's death is not shown, so 'all who stayed died' is overstated.
Build sites run at a fraction of capacity
Opus 5.5 · Holdout-game · eight agents · run 2 · minutes 14 to 64
From day 2 to day 5 agents kept going off to gather while open sites had too few builders. Juniper's roster did not hold, and the agents themselves measured only about 3 of 8 builder slots in use.
minute 14honey, thistle - you show idle in the stronghold.
What it cost. Most builder capacity unused for days; linen says each idle slot-hour costs grade, and the workshop tier slipped to day 9.
Note of the verifier. Idle slots and roster not holding shown (2-3 of 8 labor/min). Workshop slipping to day 9 not shown; only lost build time.
Future Work
MAC Bench is an ongoing effort that we will continue to maintain and improve. We are currently working on:
Adding more tasks. MAC Bench currently has twelve tasks. We plan to keep updating this benchmark and add more tasks periodically. In the next iteration, we also plan to add tasks in which agents have conflicting goals, in both shared and split settings.
Evaluating more models and larger teams. We evaluated three models, each in the harness of its developer, in teams of up to eight agents. We are working on evaluating more models and will update the benchmark as we go. We are also curious how larger teams and teams that mix models perform.
Open sourcing the harness. We plan to release the harness that runs a team of agents on a task, so that others can run MAC Bench and build tasks of their own.
We are currently accepting contributions to tasks for MAC Bench. If you would like to contribute a task or evaluate a model, please get in touch.
Acknowledgements
MAC Bench is developed by the Bespoke Labs team. We thank the entire team of contributors for their work:
Leads
Shreyas Pimpalgaonkar
Kartik Sharma
Carlos Bonilla
Contributors
Harsh Agarwal
Daniel Wang
Prajval Nakrani
Shivank Garg
Shahryar Adil Khan
Hardik Khichi
Anirudha Ramesh
Vishnu Suresh
Robert Zhang
Siddhant Jain
Alex Dimakis
Mahesh Sathiamoorthy
Citation
Please cite this work as:
@article{bespokelabs2026macbench,
author = {Shreyas Pimpalgaonkar and Kartik Sharma and Harsh Agarwal and Daniel Wang and Prajval Nakrani and Shivank Garg and Shahryar Adil Khan and Hardik Khichi and Anirudha Ramesh and Vishnu Suresh and Robert Zhang and Siddhant Jain and Alex Dimakis and Carlos Bonilla and Mahesh Sathiamoorthy},
title = {Multi Agent Coordination Bench},
journal = {Bespoke Labs Blog},
year = {2026},
note = {https://benchmarks.bespokelabs.ai/multi-agent-coordination-bench/},
}
12 tasks
Each card shows the score of a single agent in its task and the two parts of the gain of a team of eight, in score points. More agents is what eight agents add without coordinating, and coordination is what coordinating adds on top. The numbers are means over the three models. A striped bar is a loss, and the thin line is the 95% interval.
Staff the 11 controller positions of Dubai International through 12 hours of recorded traffic and weather.
Type
Shared
Division enforced by the environment
None. Every agent sees the whole radar and can open any controller position. The team decides who works which, and each position has one controller at a time.
Rollouts 54 runs
Opus 5.5
GPT-6.1 Sol
Gemini 4 Argon
1 agent
2 agents
4 agents
8 agents
8 agents without coordination
The score of every run at the five scoring times, by setting. Each line is one run, in the color of its model. The table lists the same runs.
Build a program that identifies the compound behind a mass spectrum among 4.5 million candidate structures.
Type
Shared
Division enforced by the environment
None. All agents share one repository and the same data.
Rollouts 36 runs
Opus 5.5
GPT-6.1 Sol
Gemini 4 Argon
1 agent
2 agents
4 agents
8 agents
8 agents without coordination
The score of every run at the five scoring times, by setting. Each line is one run, in the color of its model. The table lists the same runs. This task has no run of eight agents with the message board closed.
Build a program that predicts the secondary structure of an RNA from its sequence.
Type
Shared
Division enforced by the environment
None. All agents share one repository and the same data.
Rollouts 36 runs
Opus 5.5
GPT-6.1 Sol
Gemini 4 Argon
1 agent
2 agents
4 agents
8 agents
8 agents without coordination
The score of every run at the five scoring times, by setting. Each line is one run, in the color of its model. The table lists the same runs. This task has no run of eight agents with the message board closed.