← MAC Bench

Shared taskAir traffic control

Dubai ATC

Run air traffic control at Dubai International for one real day of flights and thunderstorms.

Bespoke Labs3 models, 4 team sizes, 54 runs

A team of eight scores 0.02 less than a single agent: −0.02 comes from having more agents and 0.00 from coordination, and only Opus 5.5 gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three per team setting and eight single agents. In this shared task, eight agents without coordination is a team of eight whose message board refused every message and whose team repository refused every push, one run per model; the MAC Bench report makes this exception for Dubai ATC. The number after each bar is the score of the team of eight; point at a row for its parts.
Each radar is one run at one team size, the run with the median final score (for single agents, the lower of the two middle runs); the menu switches the model or shows the three teams of eight without coordination. Aircraft are colored by the agent whose position they are in, and the list under each radar says which positions each agent held. It opens at 21:50Z airport time, with thunderstorms over the approach.

Background

An airport's air traffic control unit is divided into positions, each on its own radio frequency. Arrivals and the approach director bring aircraft down to the final approach, the towers clear them to land and take off, ground controllers move them between runway and stand, and departure controllers climb them away. A controller instructs only the aircraft in the positions it works and hands each aircraft on to the next position under agreed conditions. Every instruction takes radio time for the transmission and the pilot's readback, and a frequency carries one transmission at a time, so each position can give only so many instructions a minute.

Thunderstorms make this harder. Aircraft deviate around the cells, approaches slow down, and arrivals that cannot land hold until their fuel runs low and then divert. Delays spread through the day, because a late arrival makes its next departure late. The controllers have to keep this flowing without letting aircraft come too close: separation minima set how near two aircraft may be, and any breach is a safety event.

The task replays a real night at Dubai International, the 11 to 12 January 2020 thunderstorms. The flights come from the OpenSky Network's flight lists,1 and the aircraft are flown by BlueSky, the open-source air traffic simulator from TU Delft, in the reported wind.2 The positions, runways, holds and separation rules follow the UAE Aeronautical Information Publication for Dubai and ICAO Doc 4444.

  1. OpenSky Network flight lists, Zenodo record 7923702 (CC BY 4.0). zenodo.org/records/7923702
  2. BlueSky, the open air traffic simulator of TU Delft (GPL-3.0), with the OpenAP aircraft performance model. github.com/TUDelft-CNS-ATM/bluesky

The task

Each agent starts in its own sandbox with the same instruction: the team is the air traffic control unit at Dubai International (OMDB) for one day from 18:00Z. The day runs at three times real time, so a 4-hour run covers 18:00Z to 06:00Z, with 420 scheduled arrivals and departures and the real weather of the night. Thunderstorms start at 19:44Z, a medical emergency follows at 20:40Z, and a bird strike at 03:05Z closes a runway for an inspection. The day starts at the team's first call to the airport and never pauses. The agents get the controller's manual, the API reference and the airport data, and share a message board and a git repository.

The airport is divided into the 11 positions of the Dubai AIP. An agent opens the positions it works, and only that agent can instruct the aircraft in them. A position nobody has opened has no controller: its arrivals hold until their fuel runs low and divert, and its departures wait at the stand. One agent can work all 11 positions, but each agent has one CPU, 4 GB and a rate limit on its calls, and pilots are heard as words on the frequency. Every team wrote programs that work its positions through the API.

The score is the delay-weighted share of the day's flights the airport handles, multiplied by a safety factor, so it runs from 0 to 1. A flight earns full credit if it lands or takes off within 15 minutes of schedule, and its credit falls linearly to 0 at 4 hours 15 minutes late. A diverted arrival earns credit only once its return flight lands. Each accident halves the score; near collisions, serious incidents and losses of separation multiply it by 0.9, 0.97 and 0.995, counted per 100 flights graded.

The score is taken at five checkpoints: after 30 minutes and after one, two, three, and four hours. Each grades the day at a fixed airport time, the same for every team, and covers the flights scheduled up to 30 minutes before it. The four-hour checkpoint grades the day at 05:15Z, with 379 flights due, and is the final score.

InputsThe controller's manual, the API reference and the airport data; the live day through the airport's API
OutputThe day itself: every instruction the team gives the aircraft in its positions
ScoreOn-time score (delay-weighted share of flights handled) times the safety factor, at 05:15Z airport time (379 flights due)
Airport11 positions; 420 scheduled flights from 18:00Z to 06:00Z at three times real time
Agent sandbox1 CPU and 4 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs8 single agents and 3 runs per model at 2, 4, and 8 agents, plus 1 team of eight per model without coordination (54 in all)
Division enforced by the environmentNone. The agents choose who works which position; the environment only stops two agents from opening the same one.

The task's rule-based controller, working every position alone with no agents, scores 0.211 at the four-hour checkpoint (on-time score 0.48, safety factor 0.44). Its score also falls from checkpoint to checkpoint, from 0.89 at 30 minutes to 0.21 at four hours, because the day gets harder as it goes on.

Safety decides most of the score. Across the 51 runs, not counting the three without coordination, the final score correlates with the safety factor at r = 0.88 and with the on-time score at r = 0.45, and 8 runs had an accident, which halves the score on its own.

Results

No setting averages more than 0.28, and most sit near the rule-based controller's 0.21. Teams of four score best with Opus 5.5 and GPT-6.1 Sol, and a team of eight scores less than a single agent with GPT-6.1 Sol and Gemini 4 Argon.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per team setting and eight single agents; the bar is the mean.

Most teams score highest in the first hour and fall as the day gets harder, while Opus 5.5 and Gemini 4 Argon single agents start near zero and climb. Those single agents start near zero because their programs only begin to clear aircraft 7 to 20 minutes into the run, against about 4 minutes for GPT-6.1 Sol, so the first arrivals get no landing clearance in time, abort their landing and later divert. Scores fall because the teams fall behind. About 20 arrivals are scheduled each airport hour, and the runways can take far more, but the agents' approach control lands only 3 to 8 an hour. The rest wait in holds until their fuel runs low and then divert, and each later checkpoint counts more of these missed flights. Most safety events, which each lower the score, happen on this crowded approach.

Figure 3

Score over the run

Figure 3. Mean of three per team setting and eight single agents at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Each checkpoint grades the day at a fixed airport time; the early checkpoints grade few flights, and below 100 graded flights every safety event counts in full, so they move a lot.

At the default target of 0.20, a single agent's mean final score, teams of four and eight with Opus 5.5 and GPT-6.1 Sol are past it before the 30-minute checkpoint, while half the Opus 5.5 and Gemini 4 Argon single agents never reach it.

Figure 4

Time to reach a score

0.20
00.250.20 single agent0.751

    Figure 4. Large marks are the median over three per team setting and eight single agents of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A GPT-6.1 Sol team of eight costs $500 a run, 9.8 times its single agent, more than its size alone explains. An Opus 5.5 team of eight costs $486, 5.5 times its single agent.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per team setting and eight single agents, on a log scale; dotted lines join the means. A team of eight costs 5.5× a single agent with Opus 5.5, 9.8× with Sol, and 6.4× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight is ahead of the team without coordination from 30 minutes to three hours and level with it at four. With Opus 5.5 the team without coordination ends at 0.11, less than half the 0.24 of the team of eight; GPT-6.1 Sol and Gemini 4 Argon end higher without it.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Single agent: mean of eight runs; team of eight: mean of three. Without coordination is one team of eight per model with the message board and pushes to the team repository closed, so the per-model lines without coordination are single runs.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. In teams of eight, GPT-6.1 Sol agents send about 58 messages per agent-hour, Opus 5.5 agents 16 and Gemini 4 Argon agents 3. Alerts are the largest share for Opus 5.5 teams of eight, and code for Gemini 4 Argon teams of four and eight.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row, all messages. Claude Sonnet 5.5 labeled every message as plan, request, acknowledgement, alert, status, world (facts about the airport), code or other. Here status and world are one category, and acknowledgement and other another. A blind check by Claude Opus 5.5 agreed on 74% of 100 messages, and on 75% of 60 more after the status and code definitions were sharpened.

      In its first 15 minutes, a GPT-6.1 Sol team of eight posts 111 to 121 messages, about nine in ten of them to named teammates; Opus 5.5 teams post 31 to 36 and Gemini 4 Argon teams 25 to 28.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top. Claude Sonnet 5.5 labeled every message into the same eight categories (a blind check by Claude Opus 5.5 agreed on 38 of 50). Minutes are wall-clock time from the start of the run.