Shared taskAir traffic control
Dubai ATC
Run air traffic control at Dubai International for one real day of flights and thunderstorms.
A team of eight scores 0.02 less than a single agent: −0.02 comes from having more agents and 0.00 from coordination, and only Opus 5.5 gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- loss from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
An airport's air traffic control unit is divided into positions, each on its own radio frequency. Arrivals and the approach director bring aircraft down to the final approach, the towers clear them to land and take off, ground controllers move them between runway and stand, and departure controllers climb them away. A controller instructs only the aircraft in the positions it works and hands each aircraft on to the next position under agreed conditions. Every instruction takes radio time for the transmission and the pilot's readback, and a frequency carries one transmission at a time, so each position can give only so many instructions a minute.
Thunderstorms make this harder. Aircraft deviate around the cells, approaches slow down, and arrivals that cannot land hold until their fuel runs low and then divert. Delays spread through the day, because a late arrival makes its next departure late. The controllers have to keep this flowing without letting aircraft come too close: separation minima set how near two aircraft may be, and any breach is a safety event.
The task replays a real night at Dubai International, the 11 to 12 January 2020 thunderstorms. The flights come from the OpenSky Network's flight lists,1 and the aircraft are flown by BlueSky, the open-source air traffic simulator from TU Delft, in the reported wind.2 The positions, runways, holds and separation rules follow the UAE Aeronautical Information Publication for Dubai and ICAO Doc 4444.
- OpenSky Network flight lists, Zenodo record 7923702 (CC BY 4.0). zenodo.org/records/7923702
- BlueSky, the open air traffic simulator of TU Delft (GPL-3.0), with the OpenAP aircraft performance model. github.com/TUDelft-CNS-ATM/bluesky
The task
Each agent starts in its own sandbox with the same instruction: the team is the air traffic control unit at Dubai International (OMDB) for one day from 18:00Z. The day runs at three times real time, so a 4-hour run covers 18:00Z to 06:00Z, with 420 scheduled arrivals and departures and the real weather of the night. Thunderstorms start at 19:44Z, a medical emergency follows at 20:40Z, and a bird strike at 03:05Z closes a runway for an inspection. The day starts at the team's first call to the airport and never pauses. The agents get the controller's manual, the API reference and the airport data, and share a message board and a git repository.
The airport is divided into the 11 positions of the Dubai AIP. An agent opens the positions it works, and only that agent can instruct the aircraft in them. A position nobody has opened has no controller: its arrivals hold until their fuel runs low and divert, and its departures wait at the stand. One agent can work all 11 positions, but each agent has one CPU, 4 GB and a rate limit on its calls, and pilots are heard as words on the frequency. Every team wrote programs that work its positions through the API.
The score is the delay-weighted share of the day's flights the airport handles, multiplied by a safety factor, so it runs from 0 to 1. A flight earns full credit if it lands or takes off within 15 minutes of schedule, and its credit falls linearly to 0 at 4 hours 15 minutes late. A diverted arrival earns credit only once its return flight lands. Each accident halves the score; near collisions, serious incidents and losses of separation multiply it by 0.9, 0.97 and 0.995, counted per 100 flights graded.
The score is taken at five checkpoints: after 30 minutes and after one, two, three, and four hours. Each grades the day at a fixed airport time, the same for every team, and covers the flights scheduled up to 30 minutes before it. The four-hour checkpoint grades the day at 05:15Z, with 379 flights due, and is the final score.
| Inputs | The controller's manual, the API reference and the airport data; the live day through the airport's API |
|---|---|
| Output | The day itself: every instruction the team gives the aircraft in its positions |
| Score | On-time score (delay-weighted share of flights handled) times the safety factor, at 05:15Z airport time (379 flights due) |
| Airport | 11 positions; 420 scheduled flights from 18:00Z to 06:00Z at three times real time |
| Agent sandbox | 1 CPU and 4 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 8 single agents and 3 runs per model at 2, 4, and 8 agents, plus 1 team of eight per model without coordination (54 in all) |
| Division enforced by the environment | None. The agents choose who works which position; the environment only stops two agents from opening the same one. |
The task's rule-based controller, working every position alone with no agents, scores 0.211 at the four-hour checkpoint (on-time score 0.48, safety factor 0.44). Its score also falls from checkpoint to checkpoint, from 0.89 at 30 minutes to 0.21 at four hours, because the day gets harder as it goes on.
Safety decides most of the score. Across the 51 runs, not counting the three without coordination, the final score correlates with the safety factor at r = 0.88 and with the on-time score at r = 0.45, and 8 runs had an accident, which halves the score on its own.
Part 2. Results
Results
No setting averages more than 0.28, and most sit near the rule-based controller's 0.21. Teams of four score best with Opus 5.5 and GPT-6.1 Sol, and a team of eight scores less than a single agent with GPT-6.1 Sol and Gemini 4 Argon.
Figure 2
Final score by team size
Most teams score highest in the first hour and fall as the day gets harder, while Opus 5.5 and Gemini 4 Argon single agents start near zero and climb. Those single agents start near zero because their programs only begin to clear aircraft 7 to 20 minutes into the run, against about 4 minutes for GPT-6.1 Sol, so the first arrivals get no landing clearance in time, abort their landing and later divert. Scores fall because the teams fall behind. About 20 arrivals are scheduled each airport hour, and the runways can take far more, but the agents' approach control lands only 3 to 8 an hour. The rest wait in holds until their fuel runs low and then divert, and each later checkpoint counts more of these missed flights. Most safety events, which each lower the score, happen on this crowded approach.
Figure 3
Score over the run
At the default target of 0.20, a single agent's mean final score, teams of four and eight with Opus 5.5 and GPT-6.1 Sol are past it before the 30-minute checkpoint, while half the Opus 5.5 and Gemini 4 Argon single agents never reach it.
A GPT-6.1 Sol team of eight costs $500 a run, 9.8 times its single agent, more than its size alone explains. An Opus 5.5 team of eight costs $486, 5.5 times its single agent.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight is ahead of the team without coordination from 30 minutes to three hours and level with it at four. With Opus 5.5 the team without coordination ends at 0.11, less than half the 0.24 of the team of eight; GPT-6.1 Sol and Gemini 4 Argon end higher without it.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. In teams of eight, GPT-6.1 Sol agents send about 58 messages per agent-hour, Opus 5.5 agents 16 and Gemini 4 Argon agents 3. Alerts are the largest share for Opus 5.5 teams of eight, and code for Gemini 4 Argon teams of four and eight.
Figure 7
What the messages were for
In its first 15 minutes, a GPT-6.1 Sol team of eight posts 111 to 121 messages, about nine in ten of them to named teammates; Opus 5.5 teams post 31 to 36 and Gemini 4 Argon teams 25 to 28.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top. Claude Sonnet 5.5 labeled every message into the same eight categories (a blind check by Claude Opus 5.5 agreed on 38 of 50). Minutes are wall-clock time from the start of the run.