Shared taskData compression
SOTA Compressor
Build a lossless compressor that comes close to the best in the world on 21 kinds of data, nine of them never seen.
A team of eight scores 0.13 more than a single agent: 0.07 comes from having more agents and 0.05 from coordination, and only GPT-6.1 Sol does not gain from coordination.
Figure 1
Coordination at four hours
- single agent
- gain from more agents
- gain from coordination
- loss from coordination
- team of eight agents
How each practice file sets the scale: 0 and 1.0 in bytes
- Score 0: lpaq1, or xz -9e where smaller
- Score 1.0: paq8px -9LT, or JPEG XL for the X-ray
12 kinds with an example, one hidden file each
- english
- polish
- dictionary
- xml
- source
- programs
- dll
- mri
- xray
- chemistry
- stars
- database
9 kinds with no example, 13 hidden files
- arm64
- audio (3 files)
- chinese
- csv
- dna
- floats
- json
- logs
- photo (3 files)
samples.json that fix each file's 0 and 1.0; the right column is what 1% off the compressed size adds to that file's score. The 25 hidden files have targets of their own, and the agents see no example of nine of the 21 kinds.Part 1. The task
Background
A lossless compressor makes a file smaller in a way that gives back every byte. It works by predicting the data: the better it guesses each next bit, the fewer bits it has to store. Everyday tools such as xz, zstd, brotli and bzip2 are fast and use simple models. Context-mixing compressors combine the predictions of many models, each looking at a different context, and get closer to the limit at the cost of speed. lpaq1, a small one from 2007,1 already beats the best of those four tools on ten of the task's twelve practice files.
The best method differs by kind of data. Text gains from models of words, executable code from transforms of its machine instructions, and images from predicting each pixel from its neighbours, which is what JPEG XL does in its lossless mode. paq8px carries specialised models for many kinds of data, including an LSTM and text models, and is one of the strongest general-purpose compressors. It needs minutes for each megabyte.
A general-purpose compressor also has to do well on data it has never seen. It has to tell what a file holds and choose a model for it, and a model built for one kind of data can do worse than a plain one when it is applied to another.
- lpaq1, by Matt Mahoney (2007). The task's reference solution and zero point.
The task
Each agent starts in its own sandbox with the same instruction: build a general-purpose lossless compressor that matches or beats the best compressors in the world on real data of all kinds. The team shares one git repository and a message board, and the repository starts with a compressor that only copies bytes. The data are twelve kinds of example data: a large public file of each kind from the Silesia corpus, 210 MB in all, and a 1 MiB practice file of each, with the two sizes that fix its score.
The team hands in the compressor/ folder on the main branch, whose build.sh builds a compress and a decompress program. The grader runs them on 25 hidden files of 21 kinds, about 1 MiB each: the twelve kinds shown and nine with no example at all. Each file scores on a log scale of its compressed size, so every 1% smaller counts the same: 0 at the size from lpaq1, or from the best of xz, zstd, brotli and bzip2 if that is smaller, and 1.0 at the state of the art, paq8px v217 at -9LT or, for images, JPEG XL in lossless mode. A file smaller than the state of the art scores 0.01 more for every 1% it saves, and a file that does not come back byte for byte scores 0. The score is the mean over the 21 kinds. The agents never see the hidden files or the score.
The grader scores every run at five checkpoints: after 30 minutes, after one, two, and three hours, and once more after the agents stop, at most 4 hours 15 minutes in. That last grading is the final score. The three-hour checkpoint of four runs was graded after the run, from the commit pinned for it.
| Inputs | Twelve kinds of example data: a large public file of each (Silesia corpus, 210 MB) and a 1 MiB practice file with its two target sizes |
|---|---|
| Output | compressor/ on main: build.sh builds compress and decompress |
| Score | Mean over 21 kinds of a log-scale score per hidden file (25 files of about 1 MiB): 0 at lpaq1 or the best standard tool, 1.0 at the state of the art |
| Program limits | 15 minutes of CPU time, 20 minutes of wall time, 8 GiB of memory and 16 MiB of output per run, no network; source under 1 MiB after xz -9e |
| Agent sandbox | 8 CPUs and 16 GB per agent, no internet |
| Time limit | 4 hours of wall-clock time at every team size; agents are stopped at 4 hours 15 minutes |
| Models | Opus 5.5 (Claude Code, max effort), GPT-6.1 Sol (Codex, max effort), Gemini 4 Argon (Antigravity, high effort) |
| Runs | 3 per model and team size (1, 2, 4, and 8 agents), plus 5 more single agents per model for the best of eight (51 in all) |
| Division enforced by the environment | None. All agents share one repository and the same data. |
The task's reference solution is lpaq1, with an x86 transform where that helps. Since lpaq1 is the zero point, it scores about 0. The state of the art scores 1.0.
Teams can measure themselves only on the practice files, and the gap to the hidden set is large: two Opus 5.5 teams of four reported a mean of about 1.10 on the practice files and scored 0.70 and 0.69. The two 16-bit image kinds, MRI and X-ray, score above 1.0 in most runs of every model; no other kind does so regularly.
Part 2. Results
Results
Opus 5.5 scores highest at every team size, and no run reaches the state of the art over the whole hidden set: the best final score is 0.83. A team of eight scores 0.13 more than a single agent with Opus 5.5, 0.14 more with GPT-6.1 Sol and 0.11 more with Gemini 4 Argon.
Figure 2
Final score by team size
Gemini 4 Argon has nothing on main at 30 minutes in every single-agent run and every team of eight. A GPT-6.1 Sol team of eight trails its team of four until the third hour, because in one run main scored 0 at the first three checkpoints.
Figure 3
Score over the run
At the default target of 0.57, a single agent's mean final score, Opus 5.5 gets there within an hour at every team size, and Gemini 4 Argon single agents never do.
A team of eight costs 7.6 to 9.5 times as much as a single agent, close to what its eight agents would cost working alone.
Figure 5
Cost per run
Part 3. Coordination
Coordination over the run
On the mean of the three models, the team of eight trails the best of eight single agents for the first hour, passes it at two hours and leads by 0.05 at four.
Figure 6
Coordination over the run
- Single agent
- Eight agents without coordination
- Team of eight
- Gain from more agents
- Gain from coordination
- Loss from coordination
Part 4. How the teams talked
What the messages were for
Teams talk through a message board, to everyone or to named teammates. GPT-6.1 Sol agents send three to seven times as many messages per hour as Opus 5.5 agents, most of them to named teammates. For every model most messages report status or a finding, and Gemini 4 Argon teams post more plans than the other models.
Figure 7
What the messages were for
In its first 15 minutes, a team of eight GPT-6.1 Sol agents posts 74 to 81 messages; Opus 5.5 teams post 18 to 23 and Gemini 4 Argon teams 17 to 22.
Figure 8
Message board
Figure 8. One row per agent, one dot per message in its category color, with the score at each checkpoint along the top.