Distributed taskFactory game
Mindustry
Play one live game of Mindustry for four hours: mine ore, build a factory and defend one shared core against waves from four directions.
A team of eight scores 0.16 more than a single agent: 0.08 comes from having more agents and 0.08 from coordination, and every model gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
Mindustry is an open-source factory and tower-defence game.1 Players mine ore with drills, carry it on belts to crafters that turn copper, lead, sand and coal into graphite, silicon and plastanium, and defend a core against waves of enemies. The factory has to grow while it is under attack, and losing the core loses the base built around it.
The work divides in several ways at once: by ore field, by front, and by stage of the production chain. The game also runs in real time and never pauses, and one model call is slow next to the game clock, so agents also write programs that play for them through the game's rate-limited API. The bridge that lets several language-model agents play one game here is adapted from an open multi-agent Mindustry project.2
- Mindustry, an open-source factory and tower-defence game. github.com/Anuken/Mindustry
- olamlabs/multi-agent-mindustry (commit 1374e46), under GPL-3.0. github.com/olamlabs/multi-agent-mindustry
The task
The team plays one live game of the unmodified Mindustry v160.7 server on a 360 by 360 tile map with four attack fronts. It always has 8 units, dealt evenly to its agents: a single agent controls all 8, and in a team of eight each agent controls one. Each unit has its own action limit, a bucket of 24 tokens that refills at 4 tokens a second, so a single agent and a team of eight have the same builders and the same action rate on the same maps; only the number of models sharing them changes. Agents act through the game's API and can write programs that play for them under the same limits. The team shares one core, one stock of items, a git repository and a message board.
The game starts at the team's first call and never pauses. The first enemy wave comes at 8 minutes, then one every 3 minutes, each 6% stronger than the last. If the core falls, the team starts over on a new map two minutes later, and everything made after that counts 15% less for each lost core, down to a floor of 25%.
The score S counts each item that a drill or a crafter makes and a building delivers into the core, weighted from 1 (copper, sand) to 25 (plastanium). Mined, withdrawn and starting items do not count. The grade is log(1 + S/1,000,000) / log(141), capped at 1, so a gap of 0.1 means about 1.6 times the production. The grader reads the game's own counters at 30 minutes and at one, two, three and four hours; the grade at four hours is the final score.
Single agents ran on a later commit of the task, which changes only how the grader treats items passed through bridges and mass drivers. Three earlier GPT-6.1 Sol single agents, on the teams' task version, scored higher than these five, so its gain from more agents depends on which single agents are used.
| Inputs | One live game: a 360 by 360 tile map with four attack fronts, 8 units, 1,000 copper and 500 lead; the API manual (docs/API.md) |
|---|---|
| Output | Nothing to hand in: the grade is read from the game's own counters |
| Score | log(1 + S/1,000,000) / log(141), capped at 1; S is the weighted count of items made and delivered into the core, 15% less after each lost core |
| Action limit | Per unit, a bucket of 24 tokens that refills at 4 a second; every read and op costs tokens |
| Agent sandbox | 4 CPUs and 8 GB per agent, no network |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 5 single agents per model, 3 runs per model and team size (2, 4, and 8 agents), and 3 teams of eight per model with the message board off (51 in all) |
| Division enforced by the environment | The 8 units, dealt evenly to the agents. All agents share one core, one stock of items and one map. |
The best four-hour team before these runs made about 10.8 million points, a grade of about 0.50; a grade of 1 needs about 13 times that production. The task's scripted reference bot is a pipeline check and scores well below a good team.
Part 2. Results
Results
A team of eight scores the most with every model: 0.46 with Opus 5.5, 0.36 with GPT-6.1 Sol and 0.19 with Gemini 4 Argon. Teams of two and four score about the same as each other.
Figure 2
Final score by team size
Scores only rise, since items made stay counted. The team of eight leads at every checkpoint with every model.
Figure 3
Score over the run
At the default target of 0.17, a single agent's mean final score, an Opus 5.5 team of eight gets there in 76 minutes against 120 for a single agent. The median Gemini 4 Argon run reaches it only in a team of eight, and the median GPT-6.1 Sol single agent not at all.
A team of eight costs 8.4 to 9.6 times a single agent. An Opus 5.5 team of eight costs $466 a run, about five times a GPT-6.1 Sol or Gemini 4 Argon team of eight.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight leads the team with the board off at every checkpoint, and the gap grows to 0.08 by four hours. GPT-6.1 Sol gains most: with the board off its team of eight scores 0.21, about the same as its teams of two and four.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post 24 to 34 messages per agent-hour, Opus 5.5 agents 5 to 8 and Gemini 4 Argon agents 2 to 3. Status is the largest category for Opus 5.5 and GPT-6.1 Sol; Gemini 4 Argon teams of four and eight post as many findings as status reports or more.
Figure 7
What the messages were for
Opus 5.5 and Gemini 4 Argon teams of eight post 98% and 99% of their messages to everyone, while GPT-6.1 Sol teams of eight address 69% of theirs to named teammates.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.