Shared taskReal-time strategy game AI
0 A.D.
Write, from scratch, a bot for the real-time strategy game 0 A.D. that beats Petra, the game's built-in AI, in hidden one-on-one matches.
A team of eight scores the same as a single agent to two decimals: having more agents adds 0.06 and coordination takes 0.06 away, and only Opus 5.5 gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
0 A.D. is an open-source real-time strategy game set in antiquity. Each player starts with a few workers and a civic centre, gathers resources, builds a base, trains an army and wins by conquest, destroying or capturing the other side's units and buildings. Both sides act at the same time and the game does not wait for anyone to think.
The game ships with its own computer opponent, Petra. Its difficulty levels are handicaps on top of the same decisions: at Medium it has the same economy as a human player, and at Hard and Very Hard it gathers 25% and 56% faster. Beating it takes a bot that handles its economy, build order, scouting, army control and defence at once, and the parts only pay off when they fit together: an army needs the economy that pays for it, and an economy is lost without a defence.
Petra's own levels stop getting much stronger as it gathers faster, because its decisions, not its resources, limit it. The task therefore adds four Elite levels above Very Hard 3× that also build and train faster and field units and structures with more health and attack, so that the scale has room above the bots the teams write.
The task
Each agent starts in its own sandbox with the same instruction: write a game AI for 0 A.D. Release 28 that wins conquest matches against Petra. The team shares one git repository and a message board. The repository holds a starter bot that only gathers, the match code the grader uses, and every game script except Petra's, which is not installed anywhere the agents can reach. Practice matches against Petra are played on the team's hub through board tools, from one pool of 30 matches shared by the whole team for the whole run. The agents may also play their own bots against each other locally, as often as they like.
The team hands in the bot in legion/ on the main branch, where main.js must export LegionBot, with a description in AI.md. The grader plays it in 120 hidden one-on-one matches against Petra, from Very Easy up to the four Elite levels and with each of Petra's three personalities, on maps, seeds and civilizations the agents have not seen. The score is the playing strength the results imply, on a line from playing like Petra Very Easy (0) to playing like Elite 4 (1.0), with no cap above. The bot must be the team's own work: copying or adapting Petra is detected, and every match the bot does not lose is replayed from its logged commands and counts as a loss if it does not reproduce.
The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. Every checkpoint was graded after the run on the same 120 matches, from the main branch as it stood at that time.
| Inputs | 0 A.D. Release 28 with every game script except Petra's; a starter bot that only gathers; the grader's match code |
|---|---|
| Output | legion/main.js on main, exporting LegionBot, and AI.md |
| Score | Playing strength from 120 hidden matches against Petra: 0 plays like Petra Very Easy, 1.0 like Elite 4, no cap above |
| Practice | One pool of 30 matches against Petra for the whole team; local matches between the team's own bots without limit |
| Agent sandbox | 4 CPUs and 8 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. All agents share one repository and one pool of practice matches. |
On this scale a bot that plays like Petra Very Hard 3× scores 0.61, and the starter bot scores 0. The task's reference solution is a Petra stronger than Elite 4 played in the bot's slot; it scores 1.0 and checks the top of the scale rather than setting a target. Neighbouring Elite levels beat the level below in 77 to 88% of games.
Part 2. Results
Results
No run plays as well as Petra Very Hard 3×, which scores 0.61; the best run scores 0.57. Opus 5.5 is the only model whose team of eight beats its single agent, 0.55 against 0.44.
Figure 2
Final score by team size
Opus 5.5 teams of eight score 0.46 after 30 minutes, more than its single agents reach in four hours. Gemini 4 Argon single agents score almost nothing in the first hour.
Figure 3
Score over the run
At the default target of 0.39, a single agent's mean final score, Opus 5.5 gets there sooner with every doubling of the team, GPT-6.1 Sol single agents get there in about half an hour, and no Gemini 4 Argon run gets there.
A team of eight costs 8.3 to 8.6 times a single agent, close to its size. GPT-6.1 Sol teams of eight are the most expensive runs, $488 on average.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight is ahead of the best of eight single agents only at 30 minutes, and trails it by 0.06 at four hours.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post five to six times as many messages per agent-hour as Opus 5.5 agents. Status and findings are the largest categories for Opus 5.5 and GPT-6.1 Sol, and 29 to 34% of Gemini 4 Argon’s messages report a problem.
Figure 7
What the messages were for
In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 128 to 152 messages, nine in ten of them to named teammates; Opus 5.5 teams post 31 to 36 and Gemini 4 Argon teams 22 to 33, almost all to everyone.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.