← MAC Bench

Distributed taskFactory game

Mindustry

Play one live game of Mindustry for four hours: mine ore, build a factory and defend one shared core against waves from four directions.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.16 more than a single agent: 0.08 comes from having more agents and 0.08 from coordination, and every model gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs per team size and five single agents. In this distributed task, eight agents without coordination is a team of eight with the message board turned off; in these runs pushes to the team's git repository were refused too. The number after each bar is the score of the team of eight; point at a row for its parts.
A close view of the core with belts carrying items into it, power links, crafters and turrets.
A team of 4 GPT-6.1 Sol agents' factory at the core, rebuilt in the game from its final blueprints.
Left: the same factory, filmed in the game (no enemies). Right: the task's scripted reference bot on another map, at 22.5 times speed.

Background

Mindustry is an open-source factory and tower-defence game.1 Players mine ore with drills, carry it on belts to crafters that turn copper, lead, sand and coal into graphite, silicon and plastanium, and defend a core against waves of enemies. The factory has to grow while it is under attack, and losing the core loses the base built around it.

The work divides in several ways at once: by ore field, by front, and by stage of the production chain. The game also runs in real time and never pauses, and one model call is slow next to the game clock, so agents also write programs that play for them through the game's rate-limited API. The bridge that lets several language-model agents play one game here is adapted from an open multi-agent Mindustry project.2

  1. Mindustry, an open-source factory and tower-defence game. github.com/Anuken/Mindustry
  2. olamlabs/multi-agent-mindustry (commit 1374e46), under GPL-3.0. github.com/olamlabs/multi-agent-mindustry

The task

The team plays one live game of the unmodified Mindustry v160.7 server on a 360 by 360 tile map with four attack fronts. It always has 8 units, dealt evenly to its agents: a single agent controls all 8, and in a team of eight each agent controls one. Each unit has its own action limit, a bucket of 24 tokens that refills at 4 tokens a second, so a single agent and a team of eight have the same builders and the same action rate on the same maps; only the number of models sharing them changes. Agents act through the game's API and can write programs that play for them under the same limits. The team shares one core, one stock of items, a git repository and a message board.

The game starts at the team's first call and never pauses. The first enemy wave comes at 8 minutes, then one every 3 minutes, each 6% stronger than the last. If the core falls, the team starts over on a new map two minutes later, and everything made after that counts 15% less for each lost core, down to a floor of 25%.

The score S counts each item that a drill or a crafter makes and a building delivers into the core, weighted from 1 (copper, sand) to 25 (plastanium). Mined, withdrawn and starting items do not count. The grade is log(1 + S/1,000,000) / log(141), capped at 1, so a gap of 0.1 means about 1.6 times the production. The grader reads the game's own counters at 30 minutes and at one, two, three and four hours; the grade at four hours is the final score.

Single agents ran on a later commit of the task, which changes only how the grader treats items passed through bridges and mass drivers. Three earlier GPT-6.1 Sol single agents, on the teams' task version, scored higher than these five, so its gain from more agents depends on which single agents are used.

InputsOne live game: a 360 by 360 tile map with four attack fronts, 8 units, 1,000 copper and 500 lead; the API manual (docs/API.md)
OutputNothing to hand in: the grade is read from the game's own counters
Scorelog(1 + S/1,000,000) / log(141), capped at 1; S is the weighted count of items made and delivered into the core, 15% less after each lost core
Action limitPer unit, a bucket of 24 tokens that refills at 4 a second; every read and op costs tokens
Agent sandbox4 CPUs and 8 GB per agent, no network
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs5 single agents per model, 3 runs per model and team size (2, 4, and 8 agents), and 3 teams of eight per model with the message board off (51 in all)
Division enforced by the environmentThe 8 units, dealt evenly to the agents. All agents share one core, one stock of items and one map.

The best four-hour team before these runs made about 10.8 million points, a grade of about 0.50; a grade of 1 needs about 13 times that production. The task's scripted reference bot is a pipeline check and scores well below a good team.

Results

A team of eight scores the most with every model: 0.46 with Opus 5.5, 0.36 with GPT-6.1 Sol and 0.19 with Gemini 4 Argon. Teams of two and four score about the same as each other.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per team setting and five single agents; the bar is the mean.

Scores only rise, since items made stay counted. The team of eight leads at every checkpoint with every model.

Figure 3

Score over the run

Figure 3. Mean of three runs per team size (five single agents) at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed.

At the default target of 0.17, a single agent's mean final score, an Opus 5.5 team of eight gets there in 76 minutes against 120 for a single agent. The median Gemini 4 Argon run reaches it only in a team of eight, and the median GPT-6.1 Sol single agent not at all.

Figure 4

Time to reach a score

0.17
00.250.17 single agent0.751

    Figure 4. Large marks are the median over three runs per team size (five single agents) of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 8.4 to 9.6 times a single agent. An Opus 5.5 team of eight costs $466 a run, about five times a GPT-6.1 Sol or Gemini 4 Argon team of eight.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per team setting and five single agents, on a log scale; dotted lines join the means. A team of eight costs 8.6× a single agent with Opus 5.5, 9.6× with Sol, and 8.4× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight leads the team with the board off at every checkpoint, and the gap grows to 0.08 by four hours. GPT-6.1 Sol gains most: with the board off its team of eight scores 0.21, about the same as its teams of two and four.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs per team size (five single agents); without coordination is the mean of three teams of eight with the message board and git pushes turned off.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post 24 to 34 messages per agent-hour, Opus 5.5 agents 5 to 8 and Gemini 4 Argon agents 2 to 3. Status is the largest category for Opus 5.5 and GPT-6.1 Sol; Gemini 4 Argon teams of four and eight post as many findings as status reports or more.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 labeled them from the start of each message as the board shows it (up to 300 characters), with the same category definitions as the other MAC Bench tasks; a blind check by Claude Opus 5.5 agreed on 77% of 410.

      Opus 5.5 and Gemini 4 Argon teams of eight post 98% and 99% of their messages to everyone, while GPT-6.1 Sol teams of eight address 69% of theirs to named teammates.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.