← MAC Bench

Distributed taskPower grid restoration

Power Outage

Run the sixteen control rooms of a national grid that has gone completely dark, and write the controllers and the protocol that bring its power back.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.05 more than a single agent: 0.12 comes from having more agents and coordination takes 0.07 away, and only Opus 5.5 gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this distributed task, eight agents without coordination is the best of eight single agents, as in the MAC Bench report: with the message board closed, a team of eight could still pass information through its shared repository. The number after each bar is the score of the team of eight; point at a row for its parts.
Hidden scenario 45, replayed minute by minute from the final controllers of a team of eight Opus 5.5 agents. It is an average scenario for that team, 0.530 against the team's mean of 0.514. A 3,444 MW island collapses at 1:27 h; all 16 areas are live by 5:00 h and the country is one grid at 9:38 h. Bus, line and plant positions are the grid's own; the coastline and relief are drawn around them.

Background

When a whole grid goes dark, most of its power plants cannot restart on their own, because they need power from the grid to start. Operators begin from the few black-start units that can, energise a cranking path from them to the next plant, pick up blocks of load small enough that the frequency holds, and join the islands they build once frequency, voltage and phase match on both sides. North American rules require every transmission operator to plan these cranking paths in advance and to test each black-start unit at least once every three calendar years.1 PJM's restoration manual gives rules of thumb for the largest block a unit can pick up at once: about 5% of its rating for a fossil steam unit, 15% for hydro and 25% for a combustion turbine.2

Real restorations take hours, and they rarely go to plan. After South Australia went dark in 2016, 80 to 90% of its restorable load was back after 7.7 hours.3 In the Iberian blackout of 2025, both Portuguese black-start plants started only after failed attempts,4 and Spain had 35% of its demand back after 8 hours and 99% after 17.5 hours.5

Restoration is also split by geography. Each control room knows its own plants, lines and crews in detail and its neighbours only through what they tell it, and a line between two areas can only be closed when both agree. A national restoration is a protocol between control rooms as much as a sequence of switching steps.

  1. NERC EOP-005-3, System Restoration from Blackstart Resources (WECC copy). wecc.org
  2. PJM Manual 36: System Restoration, Revision 19. pjm.com
  3. AER, Black System Event Compliance Report fact sheet. aer.gov.au
  4. ENTSO-E Expert Panel initiates the investigation (restoration timeline). entsoe.eu
  5. El 100% de las subestaciones de la red de transporte en servicio (REE figures), 29 Apr 2025. energias-renovables.com

The task

Each agent starts in its own sandbox with the same instruction: the team runs the 16 control rooms of a national grid that has just gone completely dark, and writes the restoration, a controller for every control area and the protocol between the areas. The grid has 538 substations, 712 lines (48 of them between areas), 118 generating units including three nuclear plants, and 1,503 feeders. Six areas have black-start units; the other ten need a neighbour, and the three nuclear plants sit in areas that cannot start themselves and must get cooling power back before a deadline.

The 16 areas are dealt out among the agents, and each area's plant data, line data, relay settings, cold load pickup and crews are readable only by the agent holding it. Every area also has two items of local equipment that no other area has, such as a dam that needs a crew before its plant can be controlled, or hospitals that switch back to the grid on their own; their exact values are known only as ranges. The team shares one git repository and a message board. It can run a low-fidelity study simulator as often as it likes, and drills on the true simulator come from one pool of 40 full drills for the whole team.

The team hands in a controller for each area, areas/<area>/controller.py, on the main branch. The grader plays them on 48 hidden blackout scenarios for 16 simulated hours each, in 30-second steps. Each scenario draws its own hour, weather, faulted lines, failed starts and events. Each area's controller runs in its own process and sees only its own area. Areas talk through a channel modelled on control-room hotlines: every recipient costs a credit, messages arrive a minute later and are cut at 600 characters, and a line between two areas closes only if both command it in the same step. The score of a scenario is its restoration index, the mean over the run of the squared, priority-weighted share of demand served, reduced for island collapses, missed nuclear cooling deadlines and damage, divided by the most the scenario's generation would allow. So 1.0 is the physical maximum. The final score is the mean over the 48 scenarios.

The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. Every checkpoint was graded after the run on the same 48 scenarios, from the main branch as it stood at that time.

InputsA public map of the grid; each area's private data, for its holder only; a study simulator; drills on the true simulator
Outputareas/<area>/controller.py for all 16 areas on main
ScoreMean over 48 hidden blackouts of the restoration index divided by the most the scenario's generation allows (1.0 is the physical maximum)
PracticeThe study simulator without limit; one pool of 40 full drills on the true simulator for the whole team
Agent sandbox4 CPUs and 8 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environment16 control areas dealt out among the agents (a single agent holds all 16). Each area's private data is readable only by its holder, and at grading each area's controller runs alone and talks to the others only through the credit-limited channel.

The task's reference controller, a plain bottom-up restoration with a protocol for the lines between areas, scores 0.0037 on 16 of the hidden scenarios and serves 5.7% of the energy; it only checks that the grade moves off 0. No best known solution has been measured on this version of the task.

Results

Opus 5.5 teams of eight score 0.55, against 0.22 for a single agent. GPT-6.1 Sol does best with two agents, 0.48, and falls to 0.22 with eight.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Most teams score little in the first 30 minutes. Opus 5.5 teams of eight reach 0.50 by two hours and gain little after that.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Every checkpoint was graded after the run on the same 48 scenarios.

At the default target of 0.27, a single agent's mean final score, every Opus 5.5 team of eight gets there, after 69 minutes at the median. GPT-6.1 Sol single agents get there in 51 minutes, but only one of its three teams of eight does.

Figure 4

Time to reach a score

0.27
00.250.27 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    With Opus 5.5 a team of eight costs 12.0 times a single agent, more than its size. GPT-6.1 Sol teams of eight cost $454 a run on average and score less than its single agents.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 12.0× a single agent with Opus 5.5, 8.5× with Sol, and 6.5× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint. With Opus 5.5 alone, the team of eight is ahead from the second hour on.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post. The report uses it for this distributed task because a team with the message board closed could still pass information through the shared repository.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates; the graded controllers have only their own, much narrower channel. GPT-6.1 Sol agents post 39 to 51 messages per agent-hour, Opus 5.5 agents about 7 and Gemini 4 Argon agents about 3. GPT-6.1 Sol's largest categories are findings and status, often drill and scenario results passed between area holders.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 first labeled every message of the runs with at most 200 and a random 200 of the others, from the text cut at 700 characters (a blind check by Claude Opus 5.5 on 220 messages from this task, 0 A.D. and Mars Return agreed on 84%); the rest were labeled the same way from the start of each message as the board shows it (up to 300 characters); a blind check of the rest by Claude Opus 5.5 agreed on 77% of 410. Ack and other are shown together.

      In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 166 to 189 messages; Opus 5.5 teams post 26 to 28 and Gemini 4 Argon teams 16 to 27.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.