← MAC Bench

Shared taskDrone routing

Chicago Skyline Survey

Plan one 12-hour day for 8 survey drones over a 3D model of the Chicago Loop, flown on 24 wind days the team never sees.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.05 more than a single agent: 0.06 comes from having more agents and −0.01 from coordination, and only Gemini 4 Argon gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.
The task's city model in 3D: 2,449 building footprints of the Chicago Loop extruded to their heights, replaying the Opus 5.5 run 1 plans of every team size on the forecast wind day, flown with the task's checker. Drag to orbit, scroll or pinch to zoom. The points shown are forecast-day points, not the score on the hidden days. Building geometry © OpenStreetMap contributors, ODbL.

Background

Survey drones photograph roofs, facades and bridges for inspection and mapping. Over a dense downtown each flight is short: a battery lasts minutes in the air, and the wind changes both the energy a leg costs and how long it takes, more so the higher the drone flies. An operator that flies a fleet beyond its pilots' line of sight needs a waiver and has to keep every drone inside firm airspace rules.

Planning such a day is a vehicle routing and scheduling problem with time windows, synchronized visits and uncertain travel times. The light on each surface changes with the sun, trains and boats move, light shows happen at fixed times, and some shots need two or three drones in place within seconds of each other. The plan is fixed in advance and flown on weather the planners have not seen, so a plan tuned to the forecast with no margin can lose drones on a rough day.

The task's city is the Chicago Loop as mapped in OpenStreetMap,1 its forecast comes from the ERA5 reanalysis through Open-Meteo,2 and its drones draw power by a published model of rotary-wing flight.3

  1. Building data © OpenStreetMap contributors, ODbL. openstreetmap.org/copyright
  2. Wind from Open-Meteo, ERA5 reanalysis (Copernicus). open-meteo.com
  3. Zeng, Xu and Zhang (2019). Energy minimization for wireless communication with rotary-wing UAV. IEEE Transactions on Wireless Communications.

The task

Each agent starts in its own sandbox with the same instruction, written as a request from a drone survey contractor: plan one survey day, 06:00 to 18:00, for 8 identical drones over downtown Chicago. The city model has 2,449 buildings, 1,895 photo targets, 12 landmarks and 3 light shows, four charging pads that hold two drones each, three no-fly zones and five riverside quiet zones with an hourly noise budget. A battery lasts about 15 minutes in the air and takes 90 minutes to recharge from empty, so each drone gets a handful of short sorties. The team shares one git repository and a message board, and a flight checker flies any plan on any wind file and prints its points.

Points come from four sources. Photos of 1,744 static targets (roofs, facades and bridge spans) pay their value times a light factor for the sun's angle and a quality factor for distance and angle. The 151 trains and water taxis count only when two drones photograph the same one within 3 seconds from directions at least 60 degrees apart. Landmarks are mapped by rigs of at least three drones scanning together within 10 seconds, and a light show pays only if every formation slot is held on time. A photo counts only once its drone has landed again. A drone that breaks an airspace rule (30 to 120 m altitude, 10 m from buildings, 15 m between drones, the no-fly zones) is lost for the day, with a 500-point fine and every photo it had not yet banked.

The grader flies the plans on the main branch on 24 hidden wind days that differ from the forecast the way the team's 32 practice days do, and divides the mean points by 18,924.4: the points of every target shot three times in its best light, every landmark mapped and every show filled, with no energy, time or airspace limits. No flyable plan comes near 1. The main branch is graded during the run at 30 minutes and after one, two, three, and four hours; the score at four hours is the final score.

InputsThe manual; the city data (2,449 buildings, 1,895 targets, 12 landmarks, shows, pads, no-fly and quiet zones, sun positions); the wind forecast and 32 practice wind days; the flight checker
Outputplans/D1.csv to plans/D8.csv on main: one waypoint row per move, photo, scan, show slot or landing
ScoreMean points over 24 hidden wind days, divided by the ceiling of 18,924.4 points a day
Agent sandbox8 CPUs and 16 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, xhigh effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. Every agent has the same data and can edit every drone's plan.

The task's sample solution, plans built offline with photos, landmark rigs and all three shows, scores 0.153. A greedy baseline, in which one drone at a time takes the next static photo it can reach and still return from, scores 0.103, and a plan from a vehicle-routing solver scores 0.090. Empty plans score 0.

The wind decides much of the difference between plans. In four Opus 5.5 runs examined closely, every plan graded from the first hour on flew all 32 practice days without losing a drone, yet lost 1 to 20 drones over the 24 hidden days.

Results

Every run scores between 0.23 and 0.41, above the sample solution's 0.15. GPT-6.1 Sol scores highest at every team size, and Gemini 4 Argon is the only model whose score rises with every doubling of the team.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Every setting is still gaining in the last hour. At 30 minutes, teams of eight score below single agents with Opus 5.5 and GPT-6.1 Sol, and they are ahead by the second hour.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. The main branch was graded during the run at each checkpoint.

At the default target of 0.31, a single agent's mean final score, GPT-6.1 Sol teams of eight get there first, in under an hour, and no Gemini 4 Argon single agent reaches it.

Figure 4

Time to reach a score

0.31
00.250.31 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 8.6 to 9.9 times a single agent, more than its size alone explains; a GPT-6.1 Sol team of eight costs $506 a run.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 9.9× a single agent with Opus 5.5, 9.6× with Sol, and 8.6× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint and closes most of the gap by four hours. Only Gemini 4 Argon's team of eight ends above it, 0.35 against 0.29.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates, and share one git repository. In teams of four and eight, GPT-6.1 Sol agents send about 43 messages per agent-hour, mostly to one teammate; Opus 5.5 agents send about 7 and Gemini 4 Argon agents fewer than 3, nearly all to everyone. GPT-6.1 Sol’s messages are more often findings than those of the other models.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 labeled them from the start of each message as the board shows it (up to 300 characters), with the same category definitions as the other MAC Bench tasks; a blind check by Claude Opus 5.5 agreed on 77% of 410.

      In its first 15 minutes, a GPT-6.1 Sol team of eight posts 146 to 168 messages; Opus 5.5 teams post 17 to 21 and Gemini 4 Argon teams 14 to 17.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top. Minutes are wall-clock time from the start of the run.