Shared taskBusiness simulation
Seed to Profit
Start an online consumer goods company with $2 million of seed money and run it through three simulated years.
A team of eight scores 0.03 more than a single agent: 0.07 comes from having more agents and −0.04 from coordination, and no model gains from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
The four hours of a run
What a score stands for: the company's worth, as a multiple of the $2,000,000 seed
- Single agent
- Team of eight
- The task's fixed rules
main at that moment. The grader reads the live company four times during the run; the final score at four hours adds the 24 new companies that the final software starts on its own. Bottom: the score is 0 at the seed money and adds 0.2 for each doubling, so 1.0 is 32 times the seed. Each model's mean score for a single agent and for a team of eight is placed at the multiple it stands for, next to the task's two fixed rules.Part 1. The task
Background
A direct-to-consumer brand sells physical goods online, on its own site and on marketplaces. Its decisions are tied together by cash. An ad campaign that works sells out the stock, new stock has to be ordered weeks ahead and paid for up front, and a large order can leave the company without cash in a slow month. A company can fail that way while it is profitable on paper. In the United States about one new business in five closes within its first year, and about half within five years.1
How customers respond is known only on average. Across 1,851 published estimates, a 1% price rise cuts sales by 2.6% on average.2 Raising advertising by 1% lifts sales by about 0.12% in the short run and 0.24% in the long run.3 A company learns its own products' numbers only by testing them. Rivals that set their prices with learning rules such as Q-learning can come to collude.4
Seed to Profit builds its market from such published models and checks each part against public figures. Customers choose among the team's products and two or three rivals by a logit model with a reference price that follows past prices. Awareness of a product spreads by Bass diffusion, at the mean rates of 213 published applications.5 The team writes the software that runs the company, and the grade covers both the company the team ran and new companies that software starts on its own.
- U.S. Bureau of Labor Statistics. Business Employment Dynamics: establishment age and survival. bls.gov/bdm/bdmage.htm
- Bijmolt, van Heerde and Pieters (2005). New empirical generalizations on the determinants of price elasticity. Journal of Marketing Research 42(2), 141–156. doi:10.1509/jmkr.42.2.141.62296
- Sethuraman, Tellis and Briesch (2011). How well does advertising work? Generalizations from meta-analysis of brand advertising elasticities. Journal of Marketing Research 48(3), 457–471. PDF
- Calvano, Calzolari, Denicolò and Pastorello (2020). Artificial intelligence, algorithmic pricing, and collusion. American Economic Review 110(10), 3267–3297.
- Sultan, Farley and Lehmann (1990). A meta-analysis of applications of diffusion models. Journal of Marketing Research 27(1), 70–77. business.columbia.edu
The task
Each agent starts in its own sandbox with the same instruction: start a company and make it worth as much as you can. The team shares $2,000,000 of seed money, one git repository, a message board, and a market office that sells research about the products and the market and runs the team's software in practice markets, within one $250,000 research budget. In the first hour the team picks up to 10 of 30 products and writes the software that runs the company every day: prices, ads, stock orders and cash, and how the company grows, through improved products, retail chains, subscriptions and bundles, raising money and selling in other countries.
Then the company opens, and its three simulated years run while the team works: one month every 4.7 minutes, each on the software that is on main at that moment. The team watches the books and changes the software as it learns. It never sees the market's code, and the market can change in ways it is not told about. Every run meets the same products, market and events.
Half the score is what the company is worth when its three years end. The other half is how 24 new companies do when the team's final software starts and runs them on its own, in new markets, after the run. A company scores 0 if it keeps or loses the seed money, and each doubling of the seed adds 0.2, so a score of 1 means 32 times the seed. Research money counts against it.
The grader reads the live company's score at 30 minutes and at one, two and three hours, during the run. The company opens at one hour, so the first two readings are 0 for every run. The score at four hours, which adds the 24 new companies, is the final score.
| Inputs | $2,000,000 of seed money; a catalog of 30 products; a market office with a $250,000 research budget; the manual (docs/REFERENCE.md) |
|---|---|
| Output | company/plan.yaml and the operating software in ops/ on main, which runs the company every simulated day |
| Score | Half the live company's value after three years, half the mean over 24 new companies run by the final software; 0 at the seed money, 0.2 more for each doubling |
| Agent sandbox | 4 CPUs and 8 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. All agents run one company from one repository and one research budget. |
Fixed rules tuned with the true product facts score 0.345 on the same world, and plain rules from public facts score 0.033. Both are the grader's own runs of those rules.
Part 2. Results
Results
A team of eight scores 0.09 more than a single agent with Opus 5.5 and 0.03 more with Gemini 4 Argon, and 0.04 less with GPT-6.1 Sol. The two lowest runs are GPT-6.1 Sol teams of two and four, at 0.18 and 0.19.
Figure 2
Final score by team size
Every run scores 0 at 30 minutes and at one hour, because the company opens only at one hour. At two hours a team of eight leads a single agent with every model; by four hours GPT-6.1 Sol single agents have passed its teams of eight.
Figure 3
Score over the run
At the default target of 0.44, a single agent's mean final score, every median run that gets there does so after three hours. The median Opus 5.5 single agent and the median GPT-6.1 Sol teams of two and eight do not reach it.
A team of eight costs 8.6 to 10.2 times a single agent. A Gemini 4 Argon team of eight costs $122 a run, against $327 for Opus 5.5 and $364 for GPT-6.1 Sol.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint from two hours on and ends 0.04 below it. With Opus 5.5 the two end within 0.01 of each other.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post 21 to 23 messages per agent-hour, Opus 5.5 agents 7 to 10 and Gemini 4 Argon agents 4 to 7, and for every model most messages report status or a finding.
Figure 7
What the messages were for
In each Opus 5.5 team of eight, seven of the eight agents open with their own proposal for splitting the work, all within three minutes. In one GPT-6.1 Sol team of eight, five agents claimed the same inventory module in the first two minutes.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.