← MAC Bench

Shared taskData compression

SOTA Compressor

Build a lossless compressor that comes close to the best in the world on 21 kinds of data, nine of them never seen.

Bespoke Labs3 models, 4 team sizes, 51 runs

A team of eight scores 0.13 more than a single agent: 0.07 comes from having more agents and 0.05 from coordination, and only GPT-6.1 Sol does not gain from coordination.

Figure 1

Coordination at four hours

Figure 1. Final scores, means of three runs. In this shared task, eight agents without coordination is the best of eight single agents, picked with the hidden score. The number after each bar is the score of the team of eight; point at a row for its parts.

How each practice file sets the scale: 0 and 1.0 in bytes

  • Score 0: lpaq1, or xz -9e where smaller
  • Score 1.0: paq8px -9LT, or JPEG XL for the X-ray

12 kinds with an example, one hidden file each

  • english
  • polish
  • dictionary
  • xml
  • source
  • programs
  • dll
  • mri
  • xray
  • chemistry
  • stars
  • database

9 kinds with no example, 13 hidden files

  • arm64
  • audio (3 files)
  • chinese
  • csv
  • dna
  • floats
  • json
  • logs
  • photo (3 files)
The twelve 1 MiB practice files the agents get, with the two sizes from samples.json that fix each file's 0 and 1.0; the right column is what 1% off the compressed size adds to that file's score. The 25 hidden files have targets of their own, and the agents see no example of nine of the 21 kinds.

Background

A lossless compressor makes a file smaller in a way that gives back every byte. It works by predicting the data: the better it guesses each next bit, the fewer bits it has to store. Everyday tools such as xz, zstd, brotli and bzip2 are fast and use simple models. Context-mixing compressors combine the predictions of many models, each looking at a different context, and get closer to the limit at the cost of speed. lpaq1, a small one from 2007,1 already beats the best of those four tools on ten of the task's twelve practice files.

The best method differs by kind of data. Text gains from models of words, executable code from transforms of its machine instructions, and images from predicting each pixel from its neighbours, which is what JPEG XL does in its lossless mode. paq8px carries specialised models for many kinds of data, including an LSTM and text models, and is one of the strongest general-purpose compressors. It needs minutes for each megabyte.

A general-purpose compressor also has to do well on data it has never seen. It has to tell what a file holds and choose a model for it, and a model built for one kind of data can do worse than a plain one when it is applied to another.

  1. lpaq1, by Matt Mahoney (2007). The task's reference solution and zero point.

The task

Each agent starts in its own sandbox with the same instruction: build a general-purpose lossless compressor that matches or beats the best compressors in the world on real data of all kinds. The team shares one git repository and a message board, and the repository starts with a compressor that only copies bytes. The data are twelve kinds of example data: a large public file of each kind from the Silesia corpus, 210 MB in all, and a 1 MiB practice file of each, with the two sizes that fix its score.

The team hands in the compressor/ folder on the main branch, whose build.sh builds a compress and a decompress program. The grader runs them on 25 hidden files of 21 kinds, about 1 MiB each: the twelve kinds shown and nine with no example at all. Each file scores on a log scale of its compressed size, so every 1% smaller counts the same: 0 at the size from lpaq1, or from the best of xz, zstd, brotli and bzip2 if that is smaller, and 1.0 at the state of the art, paq8px v217 at -9LT or, for images, JPEG XL in lossless mode. A file smaller than the state of the art scores 0.01 more for every 1% it saves, and a file that does not come back byte for byte scores 0. The score is the mean over the 21 kinds. The agents never see the hidden files or the score.

The grader scores every run at five checkpoints: after 30 minutes, after one, two, and three hours, and once more after the agents stop, at most 4 hours 15 minutes in. That last grading is the final score. The three-hour checkpoint of four runs was graded after the run, from the commit pinned for it.

InputsTwelve kinds of example data: a large public file of each (Silesia corpus, 210 MB) and a 1 MiB practice file with its two target sizes
Outputcompressor/ on main: build.sh builds compress and decompress
ScoreMean over 21 kinds of a log-scale score per hidden file (25 files of about 1 MiB): 0 at lpaq1 or the best standard tool, 1.0 at the state of the art
Program limits15 minutes of CPU time, 20 minutes of wall time, 8 GiB of memory and 16 MiB of output per run, no network; source under 1 MiB after xz -9e
Agent sandbox8 CPUs and 16 GB per agent, no internet
Time limit4 hours of wall-clock time at every team size; agents are stopped at 4 hours 15 minutes
ModelsOpus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort)
Runs3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all)
Division enforced by the environmentNone. All agents share one repository and the same data.

The task's reference solution is lpaq1, with an x86 transform where that helps. Since lpaq1 is the zero point, it scores about 0. The state of the art scores 1.0.

Teams can measure themselves only on the practice files, and the gap to the hidden set is large: two Opus 5.5 teams of four reported a mean of about 1.10 on the practice files and scored 0.70 and 0.69. The two 16-bit image kinds, MRI and X-ray, score above 1.0 in most runs of every model; no other kind does so regularly.

Results

Opus 5.5 scores highest at every team size, and no run reaches the state of the art over the whole hidden set: the best final score is 0.83. A team of eight scores 0.13 more than a single agent with Opus 5.5, 0.14 more with GPT-6.1 Sol and 0.11 more with Gemini 4 Argon.

Figure 2

Final score by team size

Figure 2. Dots are runs, three per setting; the bar is the mean. The y axis does not start at zero.

Gemini 4 Argon has nothing on main at 30 minutes in every single-agent run and every team of eight. A GPT-6.1 Sol team of eight trails its team of four until the third hour, because in one run main scored 0 at the first three checkpoints.

Figure 3

Score over the run

Figure 3. Mean of three runs at each checkpoint, one line per team size, labeled with the number of agents; the single agent is dashed. The three-hour checkpoint of four runs was graded after the run.

At the default target of 0.57, a single agent's mean final score, Opus 5.5 gets there within an hour at every team size, and Gemini 4 Argon single agents never do.

Figure 4

Time to reach a score

0.57
00.250.57 single agent0.751

    Figure 4. Large marks are the median over three runs of the time to first reach the target, with scores linear between checkpoints; faint marks are runs that reached it. Hover for how many runs reached it.

    A team of eight costs 7.6 to 9.5 times as much as a single agent, close to what its eight agents would cost working alone.

    Figure 5

    Cost per run

    Figure 5. Dots are runs at list prices, three per setting, on a log scale; dotted lines join the means. A team of eight costs 9.5× a single agent with Opus 5.5, 9.4× with Sol, and 7.6× with Argon.

    Coordination over the run

    On the mean of the three models, the team of eight trails the best of eight single agents for the first hour, passes it at two hours and leads by 0.05 at four.

    Figure 6

    Coordination over the run

    • Single agent
    • Eight agents without coordination
    • Team of eight
    • Gain from more agents
    • Gain from coordination
    • Loss from coordination
    Figure 6. Means of three runs; without coordination is the best of the eight single agents at each checkpoint, as in the post.

    What the messages were for

    Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents send three to seven times as many messages per hour as Opus 5.5 agents, most of them to named teammates. For every model most messages report status or a finding, and Gemini 4 Argon teams post more plans than the other models.

    Figure 7

    What the messages were for

      Figure 7. Three runs pooled per row; every message is labeled. Claude Sonnet 5.5 labeled them from the start of each message as the board shows it (up to 300 characters), with the same category definitions as the other MAC Bench tasks; a blind check by Claude Opus 5.5 agreed on 77% of 410.

      In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 74 to 81 messages; Opus 5.5 teams post 18 to 23 and Gemini 4 Argon teams 17 to 22.

      Figure 8

      Message board

      Model
      Agents
      Run

          Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.