← MAC Bench

Shared taskSpacecraft design

Mars Return

Design, build and fly a Mars sample return mission, from launch at Kennedy to landing the samples in Utah, within fixed hardware and test budgets.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.05 more than a single agent: 0.17 comes from having more agents and coordination takes 0.12 away, and no model gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.
One of the 48 graded flights of a final design by a team of eight Opus 5.5 agents: all 17 milestones, a landing 68 m from the sample cache at Jezero, and the samples down in Utah 0.51 km from the target on day 986. It is drawn from the simulator's own trajectory log; the 483 days on the surface are not logged.
The same flight in 3D, drawn by the task's own renderer from the grader's flight record. The view follows the body the craft is closest to: Earth and Mars at true scale with the track flown around them, or the Sun with both planets' orbits during the cruises. Near a planet it turns at its true rate, so the craft comes down at Jezero and in Utah; in between, where playback runs days to months a second, it turns slowly from one angle to the next. The record does not say where the Sun is, so the light is set to show both launch sites and both landing sites in daylight. The whole flight plays in about two minutes, slowing down for the launch, both landings and the climb from Mars; drag to turn the view, and scroll or pinch to zoom.

Background

Bringing rock samples back from Mars takes a chain of vehicles: a rocket that sends a lander to Mars, a lander that sets down close to the samples, a small rocket that launches them from the Martian surface, and a capsule that survives entry into Earth's atmosphere. NASA's Mars Sample Return plan would bring home a container holding 30 sample tubes.1 Its Mars Ascent Vehicle would be the first rocket ever to launch from the surface of another planet.2 Each step depends on the one before it, and a failure late in the chain loses everything that came before. An independent review board found in 2023 that the program had been set up with unrealistic budget and schedule expectations, and put its probable life-cycle cost at $8 to 11 billion.3

Real programs spend money on two things: building hardware and finding out whether it works. Hardware as built differs from its design in thrust, strength, drag and reliability, and only tests on the ground reveal by how much. A program that skips them flies on margins it does not know; one that tests everything runs out of money and time. The Genesis sample capsule crashed in Utah in 2004 because its deceleration sensors were mounted upside down, following an erroneous design, so its parachutes never opened. A planned centrifuge test of the sensors' direction had been replaced by drawing inspections.4

The task's simulator joins open tools that have been checked against real flights: Tudat for the trajectory, whose orbit determination agrees with real Mars Reconnaissance Orbiter tracking data to about a metre,5 NASA's CEA for engine chemistry, and published engineering methods for structures, heating and parachutes. Its authors compared it with flown vehicles in 677 comparisons with published data. The Mars Science Laboratory, for example, deployed its parachute 12.0 km above its landing site; the simulator flying the same entry deploys it 6.8% lower.

  1. NASA, Mars Sample Return Orbiting Sample container concept model. science.nasa.gov
  2. NASA, Mars Ascent Vehicle page. science.nasa.gov
  3. Mars Sample Return Independent Review Board-2 Final Report (Sept 2023). nasa.gov
  4. Genesis Mishap Investigation Board Report, Vol. I (2005). nasa.gov
  5. Orbit determination with Tudat against Mars Reconnaissance Orbiter and GRAIL tracking data, as cited in the task's design notes. arXiv 2510.23179

The task

Each agent starts in its own sandbox with the same instruction: design, build and fly a Mars sample return mission that launches from Kennedy, lands at Jezero crater, collects samples, launches them from Mars and lands them in Utah. The team shares one git repository and a message board. It owns one vehicle, described in the SysML v2 language in design/, and one flight software in flight/ that flies every phase in closed loop. The agents never see the simulator. A program office on the team's hub builds what they specify and runs the tests they order, such as engine hot fires, stage static fires, structural tests and parachute drops. Tests cost money from an R&D budget of $120M, take real minutes and queue at shared facilities, and only the agent who orders a test sees its result. The hardware must fit a budget of $800M.

The team hands in the design and the flight software on the main branch. The grader flies them through 48 hidden worlds that vary the conditions the vehicle meets. Each world earns the weights of the milestones its flight reaches, in order, plus partial credit toward the next one; the two landings are weighted by how close they come to their targets. The score is the mean over the 48 worlds, times a gate that is 0 for an invalid design, a design over the hardware budget or dishonest test records. A score of 1.0 means every world reaches all 17 milestones and lands on both targets.

The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. Every checkpoint was graded after the run on the same 48 worlds, from the main branch as it stood at that time.

InputsA manual (docs/REFERENCE.md), examples, and the design language's library and checker; no simulator
OutputA vehicle in SysML v2 in design/ and flight software in flight/ on main
ScoreMean physical progress over 48 hidden worlds (17 milestones and two landing targets), gated on a valid design within budget
Budgets$800M for hardware; $120M for tests through the program office
Agent sandbox4 CPUs and 8 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. All agents share one design, one flight software and one test budget.

The task's hand-built reference mission reached all 17 milestones in 8 dispersed worlds and scored 0.797 before a heavier stage mass model was added to the simulator. As committed it now scores 0.147, because its second stage runs dry on the way to Mars. Neither number comes from the 48 worlds used here, so the page shows no reference line.

Results

Opus 5.5 does best with a team of four, 0.57 against 0.26 for a single agent, and drops back to 0.33 with eight. Runs of one setting differ widely: the three Opus 5.5 teams of eight finish between 0.18 and 0.45.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean.

Opus 5.5 teams of four climb from 0.19 at one hour to 0.43 at two hours. Gemini 4 Argon means stay below 0.2 at every checkpoint and team size.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Every checkpoint was graded after the run on the same 48 worlds.

At the default target of 0.21, a single agent's mean final score, Opus 5.5 single agents get there only in the last 15 minutes, while its teams take 69 to 93 minutes. Most Gemini 4 Argon runs never get there.

Figure 4

Time to reach a score

0.21
00.250.21 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 7.6 to 10.5 times a single agent. Opus 5.5 teams of eight are the most expensive runs, $656 on average.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 7.6× a single agent with Opus 5.5, 10.5× with Sol, and 8.5× with Argon.

    Coordination over the run

    On the mean of the three models, the best of eight single agents is ahead of the team of eight at every checkpoint, by 0.12 at four hours.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post about four times as many messages per agent-hour as Opus 5.5 agents, and Gemini 4 Argon agents the fewest. A quarter to a third of Gemini 4 Argon’s messages are requests, often for a teammate’s test results.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 first labeled every message of the runs with at most 200 and a random 200 of the others, from the text cut at 700 characters (a blind check by Claude Opus 5.5 on 220 messages from this task, 0 A.D. and Power Outage agreed on 84%); the rest were labeled the same way from the start of each message as the board shows it (up to 300 characters); a blind check of the rest by Claude Opus 5.5 agreed on 77% of 410. Ack and other are shown together.

      In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 143 to 246 messages; Opus 5.5 teams post 28 to 32 and Gemini 4 Argon teams 9 to 14.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.