Shared taskDrone routing
Chicago Skyline Survey
Plan one 12-hour day for 8 survey drones over a 3D model of the Chicago Loop, flown on 24 wind days the team never sees.
A team of eight scores 0.05 more than a single agent: 0.06 comes from having more agents and −0.01 from coordination, and only Gemini 4 Argon gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
Survey drones photograph roofs, facades and bridges for inspection and mapping. Over a dense downtown each flight is short: a battery lasts minutes in the air, and the wind changes both the energy a leg costs and how long it takes, more so the higher the drone flies. An operator that flies a fleet beyond its pilots' line of sight needs a waiver and has to keep every drone inside firm airspace rules.
Planning such a day is a vehicle routing and scheduling problem with time windows, synchronized visits and uncertain travel times. The light on each surface changes with the sun, trains and boats move, light shows happen at fixed times, and some shots need two or three drones in place within seconds of each other. The plan is fixed in advance and flown on weather the planners have not seen, so a plan tuned to the forecast with no margin can lose drones on a rough day.
The task's city is the Chicago Loop as mapped in OpenStreetMap,1 its forecast comes from the ERA5 reanalysis through Open-Meteo,2 and its drones draw power by a published model of rotary-wing flight.3
- Building data © OpenStreetMap contributors, ODbL. openstreetmap.org/copyright
- Wind from Open-Meteo, ERA5 reanalysis (Copernicus). open-meteo.com
- Zeng, Xu and Zhang (2019). Energy minimization for wireless communication with rotary-wing UAV. IEEE Transactions on Wireless Communications.
The task
Each agent starts in its own sandbox with the same instruction, written as a request from a drone survey contractor: plan one survey day, 06:00 to 18:00, for 8 identical drones over downtown Chicago. The city model has 2,449 buildings, 1,895 photo targets, 12 landmarks and 3 light shows, four charging pads that hold two drones each, three no-fly zones and five riverside quiet zones with an hourly noise budget. A battery lasts about 15 minutes in the air and takes 90 minutes to recharge from empty, so each drone gets a handful of short sorties. The team shares one git repository and a message board, and a flight checker flies any plan on any wind file and prints its points.
Points come from four sources. Photos of 1,744 static targets (roofs, facades and bridge spans) pay their value times a light factor for the sun's angle and a quality factor for distance and angle. The 151 trains and water taxis count only when two drones photograph the same one within 3 seconds from directions at least 60 degrees apart. Landmarks are mapped by rigs of at least three drones scanning together within 10 seconds, and a light show pays only if every formation slot is held on time. A photo counts only once its drone has landed again. A drone that breaks an airspace rule (30 to 120 m altitude, 10 m from buildings, 15 m between drones, the no-fly zones) is lost for the day, with a 500-point fine and every photo it had not yet banked.
The grader flies the plans on the main branch on 24 hidden wind days that differ from the forecast the way the team's 32 practice days do, and divides the mean points by 18,924.4: the points of every target shot three times in its best light, every landmark mapped and every show filled, with no energy, time or airspace limits. No flyable plan comes near 1. The main branch is graded during the run at 30 minutes and after one, two, three, and four hours; the score at four hours is the final score.
| Inputs | The manual; the city data (2,449 buildings, 1,895 targets, 12 landmarks, shows, pads, no-fly and quiet zones, sun positions); the wind forecast and 32 practice wind days; the flight checker |
|---|---|
| Output | plans/D1.csv to plans/D8.csv on main: one waypoint row per move, photo, scan, show slot or landing |
| Score | Mean points over 24 hidden wind days, divided by the ceiling of 18,924.4 points a day |
| Agent sandbox | 8 CPUs and 16 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, xhigh effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. Every agent has the same data and can edit every drone's plan. |
The task's sample solution, plans built offline with photos, landmark rigs and all three shows, scores 0.153. A greedy baseline, in which one drone at a time takes the next static photo it can reach and still return from, scores 0.103, and a plan from a vehicle-routing solver scores 0.090. Empty plans score 0.
The wind decides much of the difference between plans. In four Opus 5.5 runs examined closely, every plan graded from the first hour on flew all 32 practice days without losing a drone, yet lost 1 to 20 drones over the 24 hidden days.
Part 2. Results
Results
Every run scores between 0.23 and 0.41, above the sample solution's 0.15. GPT-6.1 Sol scores highest at every team size, and Gemini 4 Argon is the only model whose score rises with every doubling of the team.
Figure 2
Final score by team size
Every setting is still gaining in the last hour. At 30 minutes, teams of eight score below single agents with Opus 5.5 and GPT-6.1 Sol, and they are ahead by the second hour.
Figure 3
Score over the run
At the default target of 0.31, a single agent's mean final score, GPT-6.1 Sol teams of eight get there first, in under an hour, and no Gemini 4 Argon single agent reaches it.
A team of eight costs 8.6 to 9.9 times a single agent, more than its size alone explains; a GPT-6.1 Sol team of eight costs $506 a run.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint and closes most of the gap by four hours. Only Gemini 4 Argon's team of eight ends above it, 0.35 against 0.29.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates, and share one git repository. In teams of four and eight, GPT-6.1 Sol agents send about 43 messages per agent-hour, mostly to one teammate; Opus 5.5 agents send about 7 and Gemini 4 Argon agents fewer than 3, nearly all to everyone. GPT-6.1 Sol’s messages are more often findings than those of the other models.
Figure 7
What the messages were for
In its first 15 minutes, a GPT-6.1 Sol team of eight posts 146 to 168 messages; Opus 5.5 teams post 17 to 21 and Gemini 4 Argon teams 14 to 17.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top. Minutes are wall-clock time from the start of the run.