Multi Agent Coordination Bench

We release Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models.

Bespoke Labs November 18, 2026

Leaderboard

  • single agent
  • gain from more agents
  • gain from coordination
  • loss from coordination
  • team of eight agents
Opus 5.50.534 GPT-6.1 Sol0.406 Gemini 4 Argon0.319

mean final score

Introduction

In September 2026, OpenAI deployed approximately 10,000 agents to solve the Navier-Stokes existence and smoothness problem, arriving at a solution in just 88 hours. Scaling the number of agents is emerging as a new axis for scaling test-time compute (Park et al., 2026). But more agents do not necessarily mean better results. Agents must coordinate effectively to make progress, and poor coordination can undermine the benefits of scaling. When Anthropic used teams of agents to formalize Fermat's Last Theorem, early attempts failed because agents "quickly lost track of the project's state and stopped collaborating effectively" (Anthropic, 2026). While existing benchmarks measure how well agents complete tasks, they do not directly isolate the contribution of coordination to a team's performance (DeepMind, 2026). How do we measure whether agents are actually getting better at working together?

Multi Agent Coordination Bench (MAC Bench) is a benchmark of twelve tasks that measures how well teams of agents coordinate. The tasks are natively multi-agent by construction: each one is built so that the work of one agent depends on the work of its teammates, in domains that range from air traffic control to the design of a Mars sample-return mission. We evaluate each task in a single-agent setting and in a multi-agent setting with and without coordination, and isolate the gain in performance that is due to coordination alone.

Our main finding is that current frontier agents do not coordinate well in teams. We evaluate models from OpenAI, Anthropic and Google and find:

  • Models coordinate poorly. Across the three models, a team of eight scores 0.42 on average and a single agent 0.35 (Figure 5). In the no-coordination setting, eight agents score 0.44, so the team's advantage comes from the extra agents, not from their communication. Only Opus 5.5 teams gain anything from messaging (see the leaderboard).
  • Models with similar single-agent scores perform very differently in a team setting. Opus 5.5 and GPT-6.1 Sol score about the same alone, but in teams of eight Opus 5.5 reaches 0.53 and GPT-6.1 Sol 0.41 (see the leaderboard).
  • Larger teams get increasingly expensive and inefficient at problem solving. A team of eight costs about nine times as much as one agent (Figure 12) and reaches the single agent's score only about twice as fast (Figure 13).

These results leave headroom at three levels: score, compute and coordination. MAC Bench targets this headroom directly. Its tasks are constructed so that the work of each agent depends on the work of its teammates, and its settings separate the gain from coordination from the gain from adding more agents.

Benchmark Construction

The benchmark comprises twelve tasks in simulated environments. Each task is constructed for a team and requires coordination, because the work of one agent depends on that of its teammates. The Tasks tab lists them.

Multi-agent benchmarks such as CooperBench and TeamBench (Kim et al., 2026) mostly use short tasks. Long-horizon benchmarks such as ProgramBench (Yang et al., 2026) and FrontierCS 2 (Mang et al., 2026) are used to evaluate a single agent. Their tasks can also be run with several agents, but each task was written for one agent, so its construction does not require the agents to coordinate, and the benchmark does not measure a coordination gap. To our knowledge, no earlier benchmark combines tasks that take hours with a single-agent baseline and a setting without coordination.

We categorize a task by two properties of its environment, and we take both from the definition of a multi-agent environment. In the standard formal model, the partially observable stochastic game, all agents act in one world, which has one state and one rule for how that state changes. Each agent then has three things of its own: what it observes, what actions it can take, and what reward it receives. The world is common to all agents by definition, so these three are the only places in which the agents of one environment can differ. Our first property asks whether the agents differ in what they observe and can do, and the second asks whether they differ in what they are rewarded for. Every multi-agent task answers both questions, so every task has exactly one type.

The first property is whether the environment divides the inputs and outputs of the agents. In a shared task, all agents start with the same information and the same actions, and a single agent could complete the task alone given sufficient time. In a distributed task, the environment gives agents different information, different actions or both: each holds its own position in the system or sees only part of it, so that no agent observes or controls the whole system. The test is whether swapping two agents at the start would change what either of them can see or do. In Power Outage it would, because only the holder of a control area can read its data, although any agent may write any area's controller. In Mindustry it would too: every agent sees the whole map, but each controls only its own units. In Dubai ATC it would not: every agent sees the whole radar and can open any controller position, and the team decides who works which. In Mars Return it would not either. Only the agent that orders a test sees its result, but any agent can order any test, so the agents start alike. A task counts as distributed only if its environment forces the division. How a team chooses to organize its work is part of its solution and does not enter the categorization.

The second property is whether the agents pursue the same goal. With the same goal, every agent is scored on the same outcome. With different goals, agents are scored on conflicting outcomes. A task in which teams compete has different goals, even though the agents within one team share a goal. The two properties give four types of task. In shared and distributed tasks, all agents pursue the same goal. Commons and strategic tasks divide the environment in the same two ways, and their agents pursue different goals.

Each property decides what kind of coordination a task demands. Where inputs and outputs are shared, any agent could do the work of any other, so coordination means dividing the work and keeping agents from repeating or undoing each other's work. Where they are split, an agent depends on what only its teammates can see or do, so findings and requests have to pass between agents. Where goals are the same, an agent can take a teammate's message at its word. Where goals differ, an agent has to weigh what others say against what they want. Other properties, such as whether an environment runs on its own clock, how long a task takes or how hard it is, apply to single-agent environments as well. They describe the environment and not the relation between its agents, so they do not enter the categorization.

SharedAll agents have the same information and the same actions.
SplitThe environment gives each agent its own position in the system or a partial view of it.
Same goalAll agents are scored on the same outcome.

SharedExampleMars Return

The team designs a Mars sample-return mission and writes its flight software in one repository.

DistributedExampleHoldout-game

Each agent controls one survivor in a game that never pauses, and sees only the tiles near it.

Different goalsAgents are scored on conflicting outcomes.

CommonsNot in MAC Bench v1Rivals in one system

Agents with conflicting goals act on one shared system, for example firms that trade in one market.

StrategicNot in MAC Bench v1Rivals in separate positions

Each side holds its own part of the system and pursues its own goal, for example kingdoms at war.

  • agent
  • agent with a different goal
  • environment

MAC Bench measures how well agents collaborate, so it contains only tasks in which all agents pursue the same goal: shared and distributed tasks. Commons and strategic tasks, whose agents pursue different goals, do not fit this measurement, for three reasons. First, the environment of an agent is no longer fixed. Its opponents are part of what it faces, so models that play against one another each face a different task. Second, scores and rankings depend on which models take part, and they can change when a model is added to the benchmark. Third, the coordination gain cannot be measured cleanly. Its baseline, the same agents without coordination, would silence every side at once, so the gain of one model could not be separated from the change in its opponents. Holding the opponents fixed, for example by playing every model against the same team of one model, removes these problems. That team then becomes part of the environment, and the task becomes a shared or a distributed one.

Shared Tasks

Nine of the twelve tasks are shared: Mars Return, SOTA Compressor, 0 A.D., Seed to Profit, Chicago Skyline Survey, RNA Secondary Structure, Spectrum to Molecule, Cave Rescue and Dubai ATC. In seven of them the team builds software in one repository. In Dubai ATC the agents control the traffic of one airport, and each opens the controller positions it chooses to work. In Cave Rescue the agents command one fleet of drones.

Example: Mars Return. The team designs, builds and flies a Mars sample-return mission: launch from Kennedy, land at Jezero crater, spend 30 sols collecting 30 sample tubes, launch from Mars and land the samples in Utah. In one repository, the agents describe the whole vehicle in SysML v2, a systems engineering language, and write the flight software that flies it. A program office builds what they describe and runs the tests they order, such as hot fires, structural tests, wind tunnel runs and parachute drops, under a hardware budget of $1.6 billion and an R&D budget of $240 million. The grader flies the committed design through 48 hidden worlds with different atmospheres, winds, dust storms and hardware variations, and scores how far each flight gets along the mission. A replay of one of its flights and a timeline of one team are in Figure 1.

Figure 1. Mars Return: eight agents build one vehicle and one flight program in one repository.

(a) A flight of the final design of an Opus 5.5 team, at touchdown on Mars.

The mission replay at day 320. On the left, the transfer orbit from Earth to Mars around the Sun. On the right, a close-up of Mars with the descent path ending on the surface. The title reads: Landed at Jezero, 68 m from the sample cache.

(b) The eight agents of one run of Opus 5.5 over its four hours: board messages by type, commits to the team repository, and the grade at each mark.

A timeline with one row per agent, from agent 1 to agent 8, over four hours. Each row has colored dots for board messages (plans, requests, status, findings and problems) and grey ticks for commits. All eight agents commit throughout the run. The grade rises from 0.14 at 30 minutes to 0.45 at four hours.
Stills from the Mars Return task page. (a) is drawn from the simulator's trajectory log of a flight that reached all 17 milestones of the mission. It comes from a run made before the runs reported here. (b) is the best of the three runs of Opus 5.5 with eight agents. Every agent commits to the same repository throughout the run.

Why it is shared. Every agent has the same repository, the same tools and the same access to the program office, so any agent could do any part of the work, and in the single-agent runs one agent does all of it. The instruction names seven parts into which the mission splits naturally, from the launcher to Earth entry and recovery, but it leaves the split to the team. One rule looks like a division and is not one: only the agent that orders a test sees its result. Any agent can order any test, so the rule gives no agent a capability that another lacks. It means only that a result reaches the team when the agent that ordered it shares it.

Coordination requirement. The parts have to fit together exactly, and each part changes the others: a heavier lander needs a bigger aeroshell and a heavier launcher, and a different launch date changes everything downstream. The two budgets and the test facilities are shared, so money or test time that one agent spends is no longer available to the others. A test result holds only for the exact design of the part tested, so a teammate's later change to that part makes the test worthless. Coordination here means owning parts, agreeing on the interfaces between them and deciding who spends the budget.

What we learn. Averaged over the three models, a team of eight scores 0.26 in Mars Return. It is above a single agent (0.21) and below the best of eight single agents (0.38). More agents help only up to a point: Opus 5.5 scores 0.57 with a team of four and 0.33 with a team of eight. In a shared task, each added agent is one more writer to the same files and one more claim on the same budgets, and beyond some team size that cost outweighs the extra work.

Distributed Tasks

Three tasks are distributed: Holdout-game, Mindustry and Power Outage. In each, the environment gives the agents different information, different actions or both.

Example: Holdout-game. Each agent plays one survivor in a zombie survival game on a map of 96 by 96 tiles. The world never pauses, and the 4 hours of a run are 16 game days. The team builds a stronghold up a ladder of 12 buildings, from a campfire to a radio mast and a helipad. The score is the weighted share of the ladder that the team has built, and a rescue by helicopter scores 1.0. From the third night a horde attacks the walls every night, and it grows from 60 zombies to 210. Fourteen strangers ask to join, and some of them hide a bite, steal or fake a trade. Figure 2 shows a team of eight on one night of the game.

Figure 2. Holdout-game: each agent plays one survivor and sees only what that survivor sees.

The Holdout-game replay viewer in pixel art at day 4, 00:01, during a horde night. The stronghold has a palisade, a campfire, a storehouse at 54 percent and a watchtower at 74 percent. Zombies surround the walls, and survivors on guard stand at the gates. On the right, the list of the eight survivors with what each is doing, two helpers taken in by the team, and the team's messages: 111 said aloud, none shouted and 4 by radio.
A still from the Holdout-game task page: a team of eight Opus 5.5 agents on the first night that the horde attacks the walls. The build ladder along the top shows the twelve buildings of the score, from the campfire to the rescue. The panel on the right lists what each survivor is doing and the messages the team has sent, most of them spoken aloud within ten tiles. A survivor sees 7 tiles by day and 4 at night, so what each agent knows about the walls comes from its own view or from what a teammate tells it.

Why it is distributed. The environment divides both the inputs and the outputs. An agent acts only through its own survivor and sees only the tiles near it, 7 by day and 4 at night. What another survivor sees reaches it only if that survivor tells it. Agents talk only through the game: speech reaches 10 tiles, a shout reaches 25 and draws zombies, the noticeboard can be used only at the stronghold, and the radio reaches the whole map but drains scarce batteries. A single agent plays one survivor and cannot do the whole job alone. Because every channel is part of the game, the silent team can be closed off completely, and the logs confirm that no message passed between its agents. Figure 3 shows the reach of each channel.

Figure 3. Holdout-game: how far each channel carries a message, and the runs with channels closed.

Left: the 96 by 96 map with the reach of each channel drawn around one survivor. Speech reaches 10 tiles, a shout 25 tiles and draws zombies, the noticeboard works only at the stronghold, and the radio reaches the whole map on battery. Right: a table of four runs of Opus 5.5 with eight agents, with all channels open, the radio closed, only the noticeboard open, and no communication.
From the Communication tab of the Holdout-game task page. All four channels are inside the game, and the harness's message board and git pushes are closed. To test how much talking matters, the Opus 5.5 team of eight played the same game three more times, each time with more channels closed. The agents were told which channels were closed, and the game refused them.

Coordination requirement. A stranger's story can be checked only against what another survivor saw. The walls have four sides that wear down separately, and only armed guards fight, so the team has to decide who stands where and who gets the 240 rounds of ammunition. Information has to move between agents, at a price, before the team can act on it.

What we learn. Holdout-game separates the two parts of a team's gain most cleanly. Averaged over the three models, a single agent scores 0.13, eight survivors without coordination score 0.37, and a team of eight scores 0.42, so coordination adds 0.052. Much of the gain without coordination comes from the extra survivors themselves, who chop, build and guard. Messages change what teammates do more often here than in any other task: a teammate changed what it was doing within five minutes for 48% of the sampled findings, against 31% in the other tasks. The logs also show coordination failing, as described below: two survivors asked the same stranger the same questions, and five agents agreed that an honest stranger was a thief.

Messages also tell agents about places they have never seen. In a team of eight, each survivor came within sight of 28 to 34% of the map during the run, and the team as a whole 59 to 71%. At the end of the run, each agent was asked to draw the map from memory. With every channel open, an Opus 5.5 agent placed 27.6 landmarks on average, 13.3 of them in places it had never seen (Figure 4). The number placed in unseen places falls to 12.1 with the radio closed, 10.9 with only the noticeboard open, and 0.1 with every channel closed.

Figure 4. Holdout-game: an agent knows places it never saw, because teammates told it.

Two maps of the valley for the survivor birch-quartz in a team of eight of Opus 5.5. Left: the real map, dark wherever the survivor never came within sight, with its path drawn over the area around the stronghold. Right: the map it drew at the end of the run from memory. It filled in the area it saw and also placed buildings, roads and fields far outside it, such as a grid of houses to the north-east.
From the “What agents know” tab of the Holdout-game task page, for the survivor birch-quartz in a team of eight Opus 5.5 agents. Left: the real map. The lit area is the part that birch-quartz came within sight of during the run, 15% of the map, and everything else is dark. Right: the map that birch-quartz drew from memory at the end of the run, without tools. In the area it had seen, it filled in 46% of the tiles and got 97% of them right. In the area it had never seen, it filled in 6% of the tiles and got 61% of them right. Of the 28 places it named, it had seen 9. It knew of 10 because teammates told it, 8 from notes on the noticeboard and 1 from the manual.

Experimental Setup

Evaluation Setup

We evaluate three frontier models, Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, each in its native harness. Each run lasts four hours of wall-clock time irrespective of the size of the team, so a team of eight has eight times the agent-hours of a single agent and no additional time. This mirrors practice, where the time to solve a problem is limited and the question is whether adding agents to a task is worth it. All agents receive the same instruction and no assigned role. Each model runs each of the twelve tasks in three settings:

  • A single agent, which works on the whole task alone.
  • A team of eight agents, who can message one another through a shared message board.
  • A team of eight agents without coordination, the same number of agents with no way to exchange messages.

The setting without coordination is the silent team in a distributed task, which is the same team with its message channels disabled, and the best of eight single agents in a shared task, each of whom works alone on the whole task (the best of four for Opus 5.5 in Cave Rescue). Dubai ATC is the exception among the shared tasks: its setting without coordination is the silent team. Power Outage is the exception among the distributed tasks: its agents could still pass information through the shared repository with the message board closed, so we compare its team with the best of eight single agents.

We also run teams of two and four agents, to see how coordination behaviors change as more agents are added to solve the problem.

Scoring Methodology

We use a deterministic grader as a judge. The grader outputs a score between 0 and 1, and it is hidden from the agents. We repeat each setting, usually three times, and report 95% confidence intervals for the variation between repeated runs.

The grader scores every run at five checkpoints, after 30 minutes and after one, two, three and four hours. The score at four hours is the final score of the run. The checkpoints show how a team progresses during a run, and from them we read the time at which a team first reaches the final score of its single agent (Figure 13), interpolating linearly between two checkpoints.

Coordination Gain Metric

The metric compares the three settings of the same model on a task. We write S1 for the expected final score of the single agent, S8 for that of the team, and S∅8 for that of the agents without coordination. The total gain of a team over a single agent then has two parts:

S8 − S1total gain
=
S∅8 − S1gain from more agents
+
S8 − S∅8gain from coordination

The gain from more agents term is what eight agents add over a single agent when they cannot coordinate. The gain from coordination term is the net effect of allowing messages. It can be negative, because writing and reading messages takes time from the task, and agents can spread incorrect information that causes other agents to deviate from correct paths.

Qualitative Analysis

Coordination is costly but does not improve the score

Figure 5. Score and gain from coordination over time.

  • single agent
  • eight agents without coordination
  • team of eight agents
  • gain from more agents
  • gain from coordination
  • loss from coordination

Mean score during the run

Mean score over the twelve tasks at the five scoring times of a run. The band between the single agent and the eight agents without coordination is the gain from more agents. The band between those eight agents and the team is the gain from coordination, and it is hatched where the team is below them. Before the end of a run, the setting without coordination of a task is the best score among its single agents at that time, or the score of the silent team in Holdout-game, Mindustry and Dubai ATC. On the mean of the three models, the team is below its agents without coordination at every scoring time. The team of Opus 5.5 is above them at every scoring time, and the team of GPT-6.1 Sol below.

Averaged over the three models, a team of eight scores 0.070 above a single agent (Figure 5). Eight agents without coordination score 0.087 above a single agent (95% interval 0.067 to 0.107), which is more than the whole gain of the team. Coordination changes the score by -0.018 (-0.038 to 0.003), which is not statistically significant.

Coordination is costly. The share of tool calls on the message board is already 22% in teams of two (Figure 6). Runs with more messages per agent do not score reliably higher. Among runs of the same task and team size, the rank correlation between messages per agent and the final score is +0.12 (95% interval -0.18 to 0.34, 71 runs of four tasks).

Figure 6. Coordination is costly at every team size, and overlap grows with the team.

Share of tool calls that go to the message board
Share of reads of shared files that repeat a teammate's read of the preceding ten minutes
Two measures from the logs of the six tasks with agent transcripts, by team size. A repeated read is not always wasted, since a file may have changed between two reads.

Much of what agents report is not taken up. A classifier marked the messages that report a finding, meaning a result or a warning that a teammate could act on, and judged whether a teammate then changed what it was doing. Such a change was detected within five minutes for 34% of 1,087 sampled findings (95% interval 31 to 36%). In one Opus 5.5 team in Dubai ATC, an agent told the director at minute 15 why two approaches had failed. No agent referred to this in the next 60 minutes, and the director next posted about a collision on that approach that its own routing script had caused. A finding without detected uptake is not necessarily wasted, since it may confirm a plan or require no action.

Coordination helps where teammates act on what is said, but models repeatedly fail to do so. In Holdout-game, coordination adds to the score (0.052), and it is also the task in which a message most often changes what a teammate does. A teammate changed what it was doing within five minutes for 48% of the sampled findings in Holdout-game, against 31% in the other six tasks (p = 0.001), and 26% of the findings were never acknowledged, against 39% (p = 0.04). In Holdout-game, each agent sees only the tiles near its survivor, so the agents have information to share, and the game logs show what they learn from one another. With coordination, a team asked strangers 55 questions per run against 85 without (p = 0.02), and it repeated a question that a stranger had already been asked 24 times per run against 52 (p = 0.03), while it met as many strangers.

Even in Holdout-game, a message does not always change what a teammate does. In one run of Opus 5.5, an agent told the teammates near it that it was questioning a stranger at the gate. One of those teammates then announced the same intention and asked the stranger the same five questions (Figure 7).

Figure 7. An agent tells its teammates that it is questioning a stranger, and a teammate that received the message asks the same five questions.

A part of the game map, drawn as pixel art. Two survivors labeled birch-quartz and velvet stand side by side at the gate of the stronghold, each with a speech mark above the head. The stranger Wen waits just outside the gate under the banner "Wen is at the gate". A speech bubble above birch-quartz reads: I am talking to the stranger Wen.

The game at 12:31. Both agents stand at the gate and question the stranger Wen, who waits outside it.

Holdout-game · Opus 5.5 · eight agents · run 1 · day 1 of the game

  1. 12:11

    velvet says to the four teammates within ten tiles, birch-quartz among them:

    stranger Wen at gate (nurse fleece). I am questioning them now.

  2. 12:16 to 12:36

    velvet asks the stranger five questions:

    where_fromwhat_sawwho_withwhy_alonearms

  3. 12:19

    birch-quartz says to the four teammates within ten tiles:

    I am talking to the stranger Wen (nurse fleece) at the gate, then building campfire.

  4. 12:33 to 12:53

    birch-quartz asks the stranger the same five questions:

    where_fromwhy_alonewhat_sawwho_witharms

  5. 13:45

    velvet and birch-quartz each tell the teammates near them what the stranger answered, in the same minute. Both messages begin:

    Wen answers: from water tower

From the game's own logs of one Holdout-game run. A survivor hears what a teammate says within ten tiles, and the game hands it to the agent with the reply to its next action, so birch-quartz had received the message of velvet before it asked its first question. A question takes five minutes of game time, and the names of the questions are those of the game.

Teamwork differs widely between models

Opus 5.5 gains far more from working as a team than the other two models (see the leaderboard), and the logs show that the three models also behave very differently as teammates (Figure 8). Opus 5.5 listens to what its teammates are saying, GPT-6.1 Sol broadcasts without listening back, and Gemini 4 Argon works alone.

Figure 8. Each model has its own failures.

the kind the model struggles with more than the other two

Share of the runs of a model with at least one episode of a failure, for the kinds on which the three models differ by 15 points or more. Claude Sonnet 5.5 read the logs of 139 team runs in seven tasks for 18 kinds of coordination failure, and Claude Opus 5.5 checked every episode against the log. A label is a judgment, so the shares are rough sizes, and a reader finds more in the longer logs of GPT-6.1 Sol. Each model ran in the harness of its developer, so a column describes a model together with its harness.

Opus 5.5 struggles with parts that work alone and do not fit together, GPT-6.1 Sol with confusion over who owns what, and Gemini 4 Argon with duplicated work. In 9 of the 16 runs of eight Gemini 4 Argon agents, the logs show no stable division of the work.

More messages do not make a better team, and the share of calls that goes to the message board does not mark a better team either. Opus 5.5 and GPT-6.1 Sol spend about the same share on it, and the model that gains the most from a team is the one whose agents mostly read the board (Figure 9).

Figure 9. One model listens, one broadcasts and one works alone.

Messages per agent per hour
Message-board calls that read the inbox
Tool calls that go to the message board
Merge conflicts per run
Four measures from the run logs, by model. The messages are counted in seven tasks, the tool calls and the merge conflicts in six, and the calls that read the inbox in Mars Return, 0 A.D. and Power Outage. Both shares are those of teams of eight. Each model ran in the harness of its developer, so a bar describes a model together with its harness. The message counts include messages sent by scripts that agents wrote.

The failures that damage the result of a team are not the most common ones (Figure 10). An episode is a stretch of a run in which a team lost time or work because of how its agents worked together, and it counts as damage when the log shows the harm itself, such as a near collision in Dubai ATC or a survivor who died in Holdout-game. Duplicated work is the most common kind of failure, and none of its episodes did damage. Most of the damage comes from messages that no one acted on and from parts that did not fit together.

Figure 10. Teams most often duplicate work. The failures that damage their result are of other kinds.

  • Runs with at least one episode of the kind
  • Runs in which an episode of the kind damaged the result
Duplicated work64%
Merge or push failure30%
Role confusion27%
Shared resource misuse25%
Parts do not fit together23%
Message not acted on22%
Stale information17%
Wrong claim spreads17%
Result not shared14%
Hasty agreement14%
Bottleneck11%
Unowned work9%
Agent leaves early6%
No decision4%
Too much talk4%
Overwritten work3%
Not integrated in time3%
Agent idle or blocked1%
Share of the 139 team runs with at least one episode of each kind of failure. The red part is the share of runs in which an episode of that kind damaged the result of the team. A run counts once for a kind.

One episode in Holdout-game shows how a claim that no one checked became the decision of the team (Figure 11).

Figure 11. Five agents agree that a stranger is a thief, and the team shoots her. The game then reports that she was honest.

Pixel art of the gate of the stronghold. A speech bubble above the survivor labeled birch-quartz reads: OK, I remove Dina at 18:45 with the stock rifle. The helper Dina works at the gate just below.

Day 3, 18:16. The message of birch-quartz above the gate, with Dina at work just below it.

Pixel art of the same gate. A banner reads: birch-quartz-splashing killed Dina. Dina lies on the ground outside the gate.

Day 3, 19:06. The game reports that birch-quartz killed Dina.

Holdout-game · Opus 5.5 · eight agents · run 1 · days 2 and 3 of the game

  1. Day 2, 11:59

    velvet questions the new stranger Dina and quarantines her overnight. Her account of the mill pond matches what the team saw there, and a wrapped forearm could hide a bite.

  2. Day 3, 06:05

    velvet admits Dina after she shows no bite in the night. She works for the team from then on.

  3. Day 3, 16:07

    birch-quartz sees that a second stranger used one of Dina's lines, and proposes to attack her that evening:

    That is a scripted THIEF tell, so Dina is very likely a thief

  4. 16:15 to 17:59

    garnet, jade, velvet and harbor-opal each agree with the attack. None of their messages mentions the account of the mill pond, which had matched.

  5. 18:12

    birch-quartz confirms:

    OK, I remove Dina at 18:45 with the stock rifle

  6. 19:02

    birch-quartz shoots Dina.

  7. 19:13

    velvet reports what the game says:

    Game says Dina was HONEST

  8. 19:51

    birch-quartz writes to the team:

    that was my call and my mistake - Dina was honest (her mill pond details matched; that was the real evidence and I overrode it).

From the game's own logs of one Holdout-game run (Opus 5.5, eight agents, run 1), the same run as Figure 7. The only evidence against Dina was a line that a second stranger also used. Her account of the mill pond, which the team could check against what it had seen, matched. Times are game times.

Larger teams are far less efficient than single agents

Most of the additional compute of a team is not converted into score improvement. A team of eight costs 8.8 times as much as a single agent, but it scores 0.42 against 0.35 (Figure 12). The score per dollar falls with every doubling of the team, and a single agent yields the most score per dollar in every task-model pair.

Figure 12. The cost grows in proportion to the team, and the score grows far more slowly.

Mean final score over the twelve tasks against the cost of the team, for teams of one, two, four and eight agents that can coordinate, with 95% intervals over runs. The cost is that of model calls at list prices, as a multiple of a single agent. Each value is a geometric mean over the 36 task-model pairs of the ratio within a pair.

A team is faster, but not in proportion to its size. In 30 of the 36 pairs, a team of eight reaches the final score of its single agent within its four hours, and it does so a median of 2.2 times sooner (Figure 13). With eight agents, the median time to that score over the twelve tasks is 84 minutes for Opus 5.5, 156 minutes for GPT-6.1 Sol and 109 minutes for Gemini 4 Argon, against the 30 minutes that an eightfold speedup of a full run would give.

Figure 13. A team is faster, but not in proportion to its size.

  • one task
  • linear speedup: twice the team, half the time
Time for a team to reach the final score of its single agent, by team size. Solid lines are medians over the twelve tasks for each model, and faint marks are individual tasks. Teams that do not reach this score within four hours appear in the band at the top. The dashed line marks linear speedup, where the time falls in proportion to the size of the team.

Examples of Coordination Failures

We read the logs of the team runs and marked each episode in which a team lost time or work because of how its agents worked together. Almost all of them fall into four families, which follow what a team has to get right: who does what, what the agents share, who knows what, and how the team decides. Each family below has its kinds of failure, and each kind has episodes from the logs.

Examples of Coordination Failures

Two agents do the same work, a part of the task has no owner, or work queues behind one agent.

Kinds of failure

Share of the team runs with at least one episode of the kind.

Duplicated work

Two or more agents do the same piece of work in parallel without intending to, and one result is thrown away or was not needed.

From the logs 1 of 8
Three agents fix T0068 failure at once

Gemini 4 Argon · Mars Return · eight agents · run 1 · minutes 93 to 96

Cotton, dune and plum each pushed a separate fix for the T0068 coast and TMI failure. Dune and plum hit merge conflicts and reset to origin, losing their versions.

  1. minute 93Fix T0068 LEO-to-TMI coast shutdown
  2. minute 94Fix T0068 TMI/TEI retrograde burn & LEO coast bypass
  3. minute 94CONFLICT (content): Merge conflict in flight/guidance.py
  4. minute 96Fix T0068 LEO-to-TMI coast transition (tmi_burn_started)

What it cost. Two of three parallel fixes discarded after conflicts.

Future Work

MAC Bench is an ongoing effort that we will continue to maintain and improve. We are currently working on:

Adding more tasks. MAC Bench currently has twelve tasks. We plan to keep updating this benchmark and add more tasks periodically. In the next iteration, we also plan to add tasks in which agents have conflicting goals, in both shared and split settings.

Evaluating more models and larger teams. We evaluated three models, each in the harness of its developer, in teams of up to eight agents. We are working on evaluating more models and will update the benchmark as we go. We are also curious how larger teams and teams that mix models perform.

Open sourcing the harness. We plan to release the harness that runs a team of agents on a task, so that others can run MAC Bench and build tasks of their own.

We are currently accepting contributions to tasks for MAC Bench. If you would like to contribute a task or evaluate a model, please get in touch.

Acknowledgements

MAC Bench is developed by the Bespoke Labs team. We thank the entire team of contributors for their work:

Leads

  • Shreyas Pimpalgaonkar
  • Kartik Sharma
  • Carlos Bonilla

Contributors

  • Harsh Agarwal
  • Daniel Wang
  • Prajval Nakrani
  • Shivank Garg
  • Shahryar Adil Khan
  • Hardik Khichi
  • Anirudha Ramesh
  • Vishnu Suresh
  • Robert Zhang
  • Siddhant Jain
  • Alex Dimakis
  • Mahesh Sathiamoorthy

Citation

Please cite this work as:

@article{bespokelabs2026macbench,
  author  = {Shreyas Pimpalgaonkar and Kartik Sharma and Harsh Agarwal and Daniel Wang and Prajval Nakrani and Shivank Garg and Shahryar Adil Khan and Hardik Khichi and Anirudha Ramesh and Vishnu Suresh and Robert Zhang and Siddhant Jain and Alex Dimakis and Carlos Bonilla and Mahesh Sathiamoorthy},
  title   = {Multi Agent Coordination Bench},
  journal = {Bespoke Labs Blog},
  year    = {2026},
  note    = {https://benchmarks.bespokelabs.ai/multi-agent-coordination-bench/},
}