Shared taskSpacecraft design
Mars Return
Design, build and fly a Mars sample return mission, from launch at Kennedy to landing the samples in Utah, within fixed hardware and test budgets.
A team of eight scores 0.05 more than a single agent: 0.17 comes from having more agents and coordination takes 0.12 away, and no model gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
Part 1. The task
Background
Bringing rock samples back from Mars takes a chain of vehicles: a rocket that sends a lander to Mars, a lander that sets down close to the samples, a small rocket that launches them from the Martian surface, and a capsule that survives entry into Earth's atmosphere. NASA's Mars Sample Return plan would bring home a container holding 30 sample tubes.1 Its Mars Ascent Vehicle would be the first rocket ever to launch from the surface of another planet.2 Each step depends on the one before it, and a failure late in the chain loses everything that came before. An independent review board found in 2023 that the program had been set up with unrealistic budget and schedule expectations, and put its probable life-cycle cost at $8 to 11 billion.3
Real programs spend money on two things: building hardware and finding out whether it works. Hardware as built differs from its design in thrust, strength, drag and reliability, and only tests on the ground reveal by how much. A program that skips them flies on margins it does not know; one that tests everything runs out of money and time. The Genesis sample capsule crashed in Utah in 2004 because its deceleration sensors were mounted upside down, following an erroneous design, so its parachutes never opened. A planned centrifuge test of the sensors' direction had been replaced by drawing inspections.4
The task's simulator joins open tools that have been checked against real flights: Tudat for the trajectory, whose orbit determination agrees with real Mars Reconnaissance Orbiter tracking data to about a metre,5 NASA's CEA for engine chemistry, and published engineering methods for structures, heating and parachutes. Its authors compared it with flown vehicles in 677 comparisons with published data. The Mars Science Laboratory, for example, deployed its parachute 12.0 km above its landing site; the simulator flying the same entry deploys it 6.8% lower.
- NASA, Mars Sample Return Orbiting Sample container concept model. science.nasa.gov
- NASA, Mars Ascent Vehicle page. science.nasa.gov
- Mars Sample Return Independent Review Board-2 Final Report (Sept 2023). nasa.gov
- Genesis Mishap Investigation Board Report, Vol. I (2005). nasa.gov
- Orbit determination with Tudat against Mars Reconnaissance Orbiter and GRAIL tracking data, as cited in the task's design notes. arXiv 2510.23179
The task
Each agent starts in its own sandbox with the same instruction: design, build and fly a Mars sample return mission that launches from Kennedy, lands at Jezero crater, collects samples, launches them from Mars and lands them in Utah. The team shares one git repository and a message board. It owns one vehicle, described in the SysML v2 language in design/, and one flight software in flight/ that flies every phase in closed loop. The agents never see the simulator. A program office on the team's hub builds what they specify and runs the tests they order, such as engine hot fires, stage static fires, structural tests and parachute drops. Tests cost money from an R&D budget of $120M, take real minutes and queue at shared facilities, and only the agent who orders a test sees its result. The hardware must fit a budget of $800M.
The team hands in the design and the flight software on the main branch. The grader flies them through 48 hidden worlds that vary the conditions the vehicle meets. Each world earns the weights of the milestones its flight reaches, in order, plus partial credit toward the next one; the two landings are weighted by how close they come to their targets. The score is the mean over the 48 worlds, times a gate that is 0 for an invalid design, a design over the hardware budget or dishonest test records. A score of 1.0 means every world reaches all 17 milestones and lands on both targets.
The grader scores every run at five checkpoints: after 30 minutes and after one, two, three, and four hours. The score at four hours is the final score. Every checkpoint was graded after the run on the same 48 worlds, from the main branch as it stood at that time.
| Inputs | A manual (docs/REFERENCE.md), examples, and the design language's library and checker; no simulator |
|---|---|
| Output | A vehicle in SysML v2 in design/ and flight software in flight/ on main |
| Score | Mean physical progress over 48 hidden worlds (17 milestones and two landing targets), gated on a valid design within budget |
| Budgets | $800M for hardware; $120M for tests through the program office |
| Agent sandbox | 4 CPUs and 8 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. All agents share one design, one flight software and one test budget. |
The task's hand-built reference mission reached all 17 milestones in 8 dispersed worlds and scored 0.797 before a heavier stage mass model was added to the simulator. As committed it now scores 0.147, because its second stage runs dry on the way to Mars. Neither number comes from the 48 worlds used here, so the page shows no reference line.
Part 2. Results
Results
Opus 5.5 does best with a team of four, 0.57 against 0.26 for a single agent, and drops back to 0.33 with eight. Runs of one setting differ widely: the three Opus 5.5 teams of eight finish between 0.18 and 0.45.
Figure 2
Final score by team size
Opus 5.5 teams of four climb from 0.19 at one hour to 0.43 at two hours. Gemini 4 Argon means stay below 0.2 at every checkpoint and team size.
Figure 3
Score over the run
At the default target of 0.21, a single agent's mean final score, Opus 5.5 single agents get there only in the last 15 minutes, while its teams take 69 to 93 minutes. Most Gemini 4 Argon runs never get there.
A team of eight costs 7.6 to 10.5 times a single agent. Opus 5.5 teams of eight are the most expensive runs, $656 on average.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the best of eight single agents is ahead of the team of eight at every checkpoint, by 0.12 at four hours.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post about four times as many messages per agent-hour as Opus 5.5 agents, and Gemini 4 Argon agents the fewest. A quarter to a third of Gemini 4 Argon’s messages are requests, often for a teammate’s test results.
Figure 7
What the messages were for
In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 143 to 246 messages; Opus 5.5 teams post 28 to 32 and Gemini 4 Argon teams 9 to 14.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.