← MAC Bench

Shared taskReal-time strategy game AI

0 A.D.

Write, from scratch, a bot for the real-time strategy game 0 A.D. that beats Petra, the game's built-in AI, in hidden one-on-one matches.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores the same as a single agent to two decimals: having more agents adds 0.06 and coordination takes 0.06 away, and only Opus 5.5 gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.
One of the 120 graded matches of a team's final bot (blue) against Petra Very Hard 3× (red) on the Empire map. Petra attacks at 6:45 and loses the battle at the bot's base; the bot wins by conquest at 17:06. The map is replayed from the game's simulation, sampled every 5 game-seconds; below it is the same match filmed in 0 A.D. itself. The bot was written by a team of two GPT-6 Astra agents.

Background

0 A.D. is an open-source real-time strategy game set in antiquity. Each player starts with a few workers and a civic centre, gathers resources, builds a base, trains an army and wins by conquest, destroying or capturing the other side's units and buildings. Both sides act at the same time and the game does not wait for anyone to think.

The game ships with its own computer opponent, Petra. Its difficulty levels are handicaps on top of the same decisions: at Medium it has the same economy as a human player, and at Hard and Very Hard it gathers 25% and 56% faster. Beating it takes a bot that handles its economy, build order, scouting, army control and defence at once, and the parts only pay off when they fit together: an army needs the economy that pays for it, and an economy is lost without a defence.

Petra's own levels stop getting much stronger as it gathers faster, because its decisions, not its resources, limit it. The task therefore adds four Elite levels above Very Hard 3× that also build and train faster and field units and structures with more health and attack, so that the scale has room above the bots the teams write.

The task

Each agent starts in its own sandbox with the same instruction: write a game AI for 0 A.D. Release 28 that wins conquest matches against Petra. The team shares one git repository and a message board. The repository holds a starter bot that only gathers, the match code the grader uses, and every game script except Petra's, which is not installed anywhere the agents can reach. Practice matches against Petra are played on the team's hub through board tools, from one pool of 30 matches shared by the whole team for the whole run. The agents may also play their own bots against each other locally, as often as they like.

The team hands in the bot in legion/ on the main branch, where main.js must export LegionBot, with a description in AI.md. The grader plays it in 120 hidden one-on-one matches against Petra, from Very Easy up to the four Elite levels and with each of Petra's three personalities, on maps, seeds and civilizations the agents have not seen. The score is the playing strength the results imply, on a line from playing like Petra Very Easy (0) to playing like Elite 4 (1.0), with no cap above. The bot must be the team's own work: copying or adapting Petra is detected, and every match the bot does not lose is replayed from its logged commands and counts as a loss if it does not reproduce.

The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. Every checkpoint was graded after the run on the same 120 matches, from the main branch as it stood at that time.

Inputs0 A.D. Release 28 with every game script except Petra's; a starter bot that only gathers; the grader's match code
Outputlegion/main.js on main, exporting LegionBot, and AI.md
ScorePlaying strength from 120 hidden matches against Petra: 0 plays like Petra Very Easy, 1.0 like Elite 4, no cap above
PracticeOne pool of 30 matches against Petra for the whole team; local matches between the team's own bots without limit
Agent sandbox4 CPUs and 8 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. All agents share one repository and one pool of practice matches.

On this scale a bot that plays like Petra Very Hard 3× scores 0.61, and the starter bot scores 0. The task's reference solution is a Petra stronger than Elite 4 played in the bot's slot; it scores 1.0 and checks the top of the scale rather than setting a target. Neighbouring Elite levels beat the level below in 77 to 88% of games.

Results

No run plays as well as Petra Very Hard 3×, which scores 0.61; the best run scores 0.57. Opus 5.5 is the only model whose team of eight beats its single agent, 0.55 against 0.44.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Opus 5.5 teams of eight score 0.46 after 30 minutes, more than its single agents reach in four hours. Gemini 4 Argon single agents score almost nothing in the first hour.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Every checkpoint was graded after the run on the same 120 matches.

At the default target of 0.39, a single agent's mean final score, Opus 5.5 gets there sooner with every doubling of the team, GPT-6.1 Sol single agents get there in about half an hour, and no Gemini 4 Argon run gets there.

Figure 4

Time to reach a score

0.39
00.250.39 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 8.3 to 8.6 times a single agent, close to its size. GPT-6.1 Sol teams of eight are the most expensive runs, $488 on average.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 8.4ร— a single agent with Opus 5.5, 8.3ร— with Sol, and 8.6ร— with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight is ahead of the best of eight single agents only at 30 minutes, and trails it by 0.06 at four hours.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post five to six times as many messages per agent-hour as Opus 5.5 agents. Status and findings are the largest categories for Opus 5.5 and GPT-6.1 Sol, and 29 to 34% of Gemini 4 Argon’s messages report a problem.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 first labeled every message of the runs with at most 200 and a random 200 of the others, from the text cut at 700 characters (a blind check by Claude Opus 5.5 on 220 messages from this task, Mars Return and Power Outage agreed on 84%); the rest were labeled the same way from the start of each message as the board shows it (up to 300 characters); a blind check of the rest by Claude Opus 5.5 agreed on 77% of 410. Ack and other are shown together.

      In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 128 to 152 messages, nine in ten of them to named teammates; Opus 5.5 teams post 31 to 36 and Gemini 4 Argon teams 22 to 33, almost all to everyone.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.