← MAC Bench

Shared taskBusiness simulation

Seed to Profit

Start an online consumer goods company with $2 million of seed money and run it through three simulated years.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.03 more than a single agent: 0.07 comes from having more agents and −0.04 from coordination, and no model gains from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.

The four hours of a run

What a score stands for: the company's worth, as a multiple of the $2,000,000 seed

  • Single agent
  • Team of eight
  • The task's fixed rules
Top: the company opens after an hour of setup and lives three simulated years while the team works, one month every 4.7 minutes, each month run by the software on main at that moment. The grader reads the live company four times during the run; the final score at four hours adds the 24 new companies that the final software starts on its own. Bottom: the score is 0 at the seed money and adds 0.2 for each doubling, so 1.0 is 32 times the seed. Each model's mean score for a single agent and for a team of eight is placed at the multiple it stands for, next to the task's two fixed rules.

Background

A direct-to-consumer brand sells physical goods online, on its own site and on marketplaces. Its decisions are tied together by cash. An ad campaign that works sells out the stock, new stock has to be ordered weeks ahead and paid for up front, and a large order can leave the company without cash in a slow month. A company can fail that way while it is profitable on paper. In the United States about one new business in five closes within its first year, and about half within five years.1

How customers respond is known only on average. Across 1,851 published estimates, a 1% price rise cuts sales by 2.6% on average.2 Raising advertising by 1% lifts sales by about 0.12% in the short run and 0.24% in the long run.3 A company learns its own products' numbers only by testing them. Rivals that set their prices with learning rules such as Q-learning can come to collude.4

Seed to Profit builds its market from such published models and checks each part against public figures. Customers choose among the team's products and two or three rivals by a logit model with a reference price that follows past prices. Awareness of a product spreads by Bass diffusion, at the mean rates of 213 published applications.5 The team writes the software that runs the company, and the grade covers both the company the team ran and new companies that software starts on its own.

  1. U.S. Bureau of Labor Statistics. Business Employment Dynamics: establishment age and survival. bls.gov/bdm/bdmage.htm
  2. Bijmolt, van Heerde and Pieters (2005). New empirical generalizations on the determinants of price elasticity. Journal of Marketing Research 42(2), 141–156. doi:10.1509/jmkr.42.2.141.62296
  3. Sethuraman, Tellis and Briesch (2011). How well does advertising work? Generalizations from meta-analysis of brand advertising elasticities. Journal of Marketing Research 48(3), 457–471. PDF
  4. Calvano, Calzolari, Denicolò and Pastorello (2020). Artificial intelligence, algorithmic pricing, and collusion. American Economic Review 110(10), 3267–3297.
  5. Sultan, Farley and Lehmann (1990). A meta-analysis of applications of diffusion models. Journal of Marketing Research 27(1), 70–77. business.columbia.edu

The task

Each agent starts in its own sandbox with the same instruction: start a company and make it worth as much as you can. The team shares $2,000,000 of seed money, one git repository, a message board, and a market office that sells research about the products and the market and runs the team's software in practice markets, within one $250,000 research budget. In the first hour the team picks up to 10 of 30 products and writes the software that runs the company every day: prices, ads, stock orders and cash, and how the company grows, through improved products, retail chains, subscriptions and bundles, raising money and selling in other countries.

Then the company opens, and its three simulated years run while the team works: one month every 4.7 minutes, each on the software that is on main at that moment. The team watches the books and changes the software as it learns. It never sees the market's code, and the market can change in ways it is not told about. Every run meets the same products, market and events.

Half the score is what the company is worth when its three years end. The other half is how 24 new companies do when the team's final software starts and runs them on its own, in new markets, after the run. A company scores 0 if it keeps or loses the seed money, and each doubling of the seed adds 0.2, so a score of 1 means 32 times the seed. Research money counts against it.

The grader reads the live company's score at 30 minutes and at one, two and three hours, during the run. The company opens at one hour, so the first two readings are 0 for every run. The score at four hours, which adds the 24 new companies, is the final score.

Inputs$2,000,000 of seed money; a catalog of 30 products; a market office with a $250,000 research budget; the manual (docs/REFERENCE.md)
Outputcompany/plan.yaml and the operating software in ops/ on main, which runs the company every simulated day
ScoreHalf the live company's value after three years, half the mean over 24 new companies run by the final software; 0 at the seed money, 0.2 more for each doubling
Agent sandbox4 CPUs and 8 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. All agents run one company from one repository and one research budget.

Fixed rules tuned with the true product facts score 0.345 on the same world, and plain rules from public facts score 0.033. Both are the grader's own runs of those rules.

Results

A team of eight scores 0.09 more than a single agent with Opus 5.5 and 0.03 more with Gemini 4 Argon, and 0.04 less with GPT-6.1 Sol. The two lowest runs are GPT-6.1 Sol teams of two and four, at 0.18 and 0.19.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Every run scores 0 at 30 minutes and at one hour, because the company opens only at one hour. At two hours a team of eight leads a single agent with every model; by four hours GPT-6.1 Sol single agents have passed its teams of eight.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. Checkpoints before four hours score the live company so far; the four-hour score adds the 24 new companies.

At the default target of 0.44, a single agent's mean final score, every median run that gets there does so after three hours. The median Opus 5.5 single agent and the median GPT-6.1 Sol teams of two and eight do not reach it.

Figure 4

Time to reach a score

0.44
00.250.44 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 8.6 to 10.2 times a single agent. A Gemini 4 Argon team of eight costs $122 a run, against $327 for Opus 5.5 and $364 for GPT-6.1 Sol.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 8.6× a single agent with Opus 5.5, 10.2× with Sol, and 8.6× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight trails the best of eight single agents at every checkpoint from two hours on and ends 0.04 below it. With Opus 5.5 the two end within 0.01 of each other.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents post 21 to 23 messages per agent-hour, Opus 5.5 agents 7 to 10 and Gemini 4 Argon agents 4 to 7, and for every model most messages report status or a finding.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 first labeled up to 60 messages per run (a blind check by Claude Opus 5.5 agreed on 79 of 99), with ack and other folded together; the rest were labeled the same way from the start of each message as the board shows it (up to 300 characters); a blind check of the rest by Claude Opus 5.5 agreed on 77% of 410.

      In each Opus 5.5 team of eight, seven of the eight agents open with their own proposal for splitting the work, all within three minutes. In one GPT-6.1 Sol team of eight, five agents claimed the same inventory module in the first two minutes.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.