By Daniel Huber · original experiment · October 22, 2026 · 13 min read
The short version
We gave the same tasks to one AI agent and to teams of four, in five set-ups. The four agents either worked alone without seeing each other, shared a notes file, or could also send each other messages and were told how to work as a team. Each team of four had the same token budget (a hard cap on the units of text the model reads and writes): four times what the single agent got. So the four team set-ups had equal compute with each other, and the single agent had a quarter of it.
The tasks were 30 research questions, answered by searching a fixed library of documents, and 30 number-theory problems, solved with a small Python tool. Each set-up ran every task twice, 600 runs in all.
On the research questions, four agents working alone, with four times the tokens, solved far more than one: 50% against 27%. Letting the four talk did not add to that: 47% with messages, 50% without. They used the channel little: most research runs had four messages in all.
What this means for anyone building agents: test a talking team against the same number of agents working independently on the same budget. For this model on these tasks, the talking team did not beat that baseline.
Verdict
Four Qwen3.8-27B agents working alone, with four times the tokens of one agent, solved more research questions than the one agent, a gain of +23.3 points. At the same budget as the isolated four, messages, a blackboard and team norms changed that by −3.3 points, with an interval that spans zero. The pre-registered verdict is ONLY PARALLELISM: four isolated agents beat one by at least 10 points, and talking did not beat isolated. This design cannot tell parallel attempts apart from simply having more tokens.
Original measurement−3.3 pts
Four agents with messages vs four working alone, same token pool, research questions (95% interval −15.0 to +6.7)
+23.3 pts
Four agents working alone vs one agent with a quarter of the tokens, research questions (+11.7 to +36.7)
0.92
Messages sent per agent in a run with messages and team norms (research questions)
60 paired runs per comparison on the primary suite, one model, teams of 4 · 95% bootstrap intervals clustered by task · McNemar p = 0.79 for talking vs alone
Five set-ups
Each set-up used the same model and tools. The four team set-ups shared the same token pool, four times the single agent’s; among them, only the ways the agents could reach each other changed.
- Single: one agent with a budget B, a quarter of a team’s.
- Isolated: four agents with a shared pool of 4 × B and no way to see each other. This is the baseline the talking team is compared with, at the same budget.
- Stigmergic: four agents that share only a blackboard, a common notes file. Stigmergy is coordination through traces left in a shared space, the way ants coordinate through scent.
- Polylogue (our codename; a polylogue is a conversation among many speakers): direct messages and topic broadcasts on top of the blackboard, plus four team norms in the prompt. The first norm is a kickoff: publish a one-line plan before starting work.
- Polylogue without norms: the same tools, without the norms, to tell the effect of the norms from the effect of the channel.
A team gives one answer. In the three set-ups with a shared space, the first claim ends the run, right or wrong, and the agents are told so. Isolated agents cannot see each other, so their run goes on until a claim passes or the pool is empty. Isolated is therefore scored two ways: by its first claim, the team answer and the main comparison, and by best-of-4 (best-of-N with N = 4: the team counts as right if any agent’s claim passes the checker). Best-of-4 is a reference, because it needs a checker that most real tasks do not have.
Tasks and budgets
There were two suites of 30 tasks, each with an answer that a program can check exactly.
The research suite comes from BrowseComp-Plus, a benchmark of hard research questions that ships with its own fixed, offline collection of 100,195 documents, so every run searches the same library and nothing drifts. The agents search it with BM25, a classic keyword-ranking search. Questions were chosen by two task properties only: the model answers correctly when handed the right documents, and the evidence is spread over many documents, which should favour a team that splits the reading. This suite carries the primary test.
The math suite is generated: number-theory problems such as sums of primes, or of Euler’s totient (for each n, the count of whole numbers up to n that share no factor with it), up to about 1011, with answers computed in code, sized so that a correct method only just fits the limits of a sandboxed Python tool. Contest problems were ruled out because the model may have seen them in training.
Each team of four had a hard token budget of 4 × B, reserved call by call against a shared pool, with messages counted against it: 320k tokens per team on research questions (B = 80k) and 152k on math (B = 38k). The single agent had B. A run also stopped at 30 minutes. A deterministic program checked every answer; no language model acted as judge.
Results
In plain terms: on research questions, four agents working alone clearly beat one, with four times the tokens; on math the gain was smaller and not significant. No way of letting the four share information brought a significant gain over working alone.
| Row | Research questions (BrowseComp-Plus) | Number-theory problems |
|---|---|---|
| Single agent | 27% | 35% |
| Isolated: 4 agents, no contact | 50% | 45% |
| Stigmergic: shared blackboard | 53% | 42% |
| Polylogue: messages and norms | 47% | 38% |
| Polylogue without norms | 45% | 38% |
| Isolated, best of 4 (reference) | 50% | 50% |
On research questions, isolated agents solved 30 of 60 runs and a single agent 16 of 60, a gain of +23.3 points (+11.7 to +36.7). The talking team solved 28 of 60, −3.3 points against isolated (−15.0 to +6.7). The pre-registered bar was a gain of at least 10 points with an interval above zero; the interval’s upper end, +6.7, lies below that bar.
The intervals are 95% bootstrap intervals: the range of the difference across many resamples of the tasks, drawn task by task so that the two seeds of a task stay together. Each comparison pairs the two set-ups on the same task and seed, 60 pairs on a suite. The McNemar test, a test for paired yes-or-no outcomes, looks only at the pairs where the two set-ups disagree: the talking team alone solved 6 of them and the isolated team alone 8 (p = 0.79). Against a single agent, isolated alone solved 14 and the single agent alone 0.
The blackboard team was highest on research questions with 32, +3.3 points over isolated (McNemar p = 0.75), not a significant difference. On math the order differed. Isolated solved 27 of 60 by its first claim and 30 by best-of-4, ahead of the blackboard team with 25, both polylogue teams with 23 and a single agent with 21. On math alone, isolated over single was +10.0 points (McNemar p = 0.15) and the talking team over single +3.3; neither is significant. Pooled over both suites, isolated over single was +16.7 points (+8.3 to +25.8). Over both suites, best-of-4 added +2.5 points to isolated’s first claim. Ending a team run at its first claim cost little: on research questions, 2 of the talking teams’ 30 claims were wrong.
Two secondary comparisons, both suites pooled: the blackboard team against the talking team, +5.0 points (−2.5 to +12.5), and the talking team with norms against the one without, +0.8 points (−5.0 to +6.7). Neither interval excludes zero.
How much the agents talked
The verdict is only meaningful if the agents actually used the channel, so the pre-registration made this a required check.
On research questions, talking agents sent 0.92 messages each on average, and no run had more than 4. In 44 of 60 runs the team sent exactly four messages in all. (In the pilot, such messages were one opening plan per agent, as the kickoff norm asked, all alike and never followed up; the main-run table records counts, not content.) A message counted as substantive if another agent’s next step cited or acted on its content, not counting numbers, document ids or names that already appear in the task. At least one was sent in 12 of 60 research runs and 23 of 60 math runs; on math, agents sent 0.53 messages each, between 0 and 6 per run.
Without the norms, agents almost never wrote: 3 messages among 240 agents on research questions, 1 among 240 on math. The blackboard went almost unused as well: stigmergic teams wrote 1 entry on research questions and 4 on math, in 60 runs each.
So this experiment tests talk as this model practised it with these prompts: a few messages at most, then four agents working largely alone.
The other hypotheses
Four further predictions were registered with the primary one. H1, H3 and H4 were not supported. H5 carries no verdict by design; its prediction held on math only.
| Prediction | Result | Outcome | |
|---|---|---|---|
| H2 (primary) | Talking team solves at least 10 points more research questions than isolated | −3.3 points (−15.0 to +6.7), p = 0.79 | ONLY PARALLELISM |
| H1, speed | Talking team reaches a verified answer in at most 0.8 × isolated’s time | 1.07 × (0.78 to 1.49), 22 research pairs both solved | Not supported |
| H3, domain | Talking helps more on research than on math | +3.3 points difference of differences, one-sided bound −10.0, p = 0.37 | Not supported |
| H4, no norms | Without norms, under 30% of agents send a substantive message, and solve rates stay within 5 points of isolated | 2 of 480 agents; −5.8 points (−13.3 to +1.7) | Not supported |
| H5, cost | Talking team costs at least 1.5 × isolated per solved task | research 1.05 × (0.77 to 1.48), math 1.51 × (1.08 to 2.22) | Met on math only; reported, no verdict |
H4 failed only on its second half. Its first half held clearly: without norms, almost no agent sent a substantive message. But the team without norms came out at −5.8 points against isolated over both suites (−13.3 to +1.7). Neither the point estimate nor the interval lies within the ±5 points the prediction allowed. H1, H2 and H3 were corrected together for multiple testing with Holm’s method, which raises each p-value to account for the others; all three came out at 1.0.
On cost, the talking team used about as many tokens per solved research question as the isolated one, and about half again as many per solved math problem. On research questions the median run used 286k tokens for isolated and 235k for the talking team, so working alone did not save tokens there. On math its median run used 132k tokens against 121k for isolated, and it solved fewer tasks.
What the pilots changed
Three pilot rounds and a few calibration runs, on extra tasks that are not part of the results, set the budgets and limits. Each change below was made before the main run and is recorded in the pre-registration. Pilot numbers are not results.
- A third suite was dropped. A designed suite of generated record collections with counting and summing questions was meant to need a split of the reading. In calibration (36 runs at B = 160k), no set-up solved a single task. One agent read 54 to 80 documents before its budget ran out, because each call is charged for the whole context. A talking team did divide the reading and covered every document of the smallest collections, then ran out of tokens while waiting for each other’s results and combining them. Coordination costs tokens, and on a task too large for one agent it left the team none to finish with.
- A kickoff norm was added. In pilot P1, the second round after P0, no agent sent a single message in 40 runs. With the kickoff norm, every talking run sent messages, but only one opening plan per agent, and none was followed up.
- The research budget was lowered. At the first budget rule, B = 160k, isolated agents solved 9 of 10 pilot research runs (5 tasks × 2 seeds), which would have left no room to show a gain. B was set to 80k, where isolated solves about 50 to 60% of pilot tasks.
One change came during the main run. On 18 October, with runs in progress and before any outcome was looked at (only whether runs had completed), the definition of a substantive message was narrowed so that it no longer counted a number, document id or name that also appears in the task. On pilot data, the old definition had counted kickoff plans that merely repeated the question’s own numbers. The analysis plan was fixed the same day, also before any outcome was seen. Both are dated in the pre-registration.
What this means
For Qwen3.8-27B on these two suites, a team of four beat one agent clearly on research questions, but with four times the tokens; this design cannot say whether parallel attempts or simply the larger budget bought the gain. Among the four team set-ups, which did have equal compute, a channel and norms brought no significant gain, and the agents used them little. Why they talked so little was not tested. Missing coordination training is one hypothesis, and a later phase can test it.
Other reports point the other way, with caveats this test was built to address. In an interview on the Dwarkesh Podcast, Noam Brown described four agents finishing some benchmarks about twice as fast at about twice the cost, with coordination learned in training and no control experiment. GitSwarm shows gains from many agents coordinating through a shared Git repository, measured against a single agent prompted to continue. On its proof benchmark, best-of-N with an oracle verifier reaches the same ceiling; what it lacks is a control at matched compute. Neither rules out that talk helps; both leave open how much of the gain parallel attempts alone would have bought.
For anyone building multi-agent systems: before paying for agents to talk, run the same number of agents independently on the same budget and take the first answer, or the best one if you have a check. That is the baseline a coordination scheme has to beat, and here it cost about the same per solved research question. With this model and these prompts, telling agents to collaborate was not enough to make them do it.
Next, phase 2 tests a designed coordinator protocol on the same model, in place of free messaging. A later phase may train coordination directly, with multi-agent reinforcement learning on a smaller model, which would test whether training is what is missing.
Method
The experiment was pre-registered before the first scored run; the pre-registration is public and its git history dates it.
The model was Qwen3.8-27B in NVFP4, a 4-bit weight format (revision 7d6f8d4), served with vLLM on one NVIDIA DGX Spark. The 600 runs (5 set-ups × 2 suites × 30 tasks × 2 seeds) ran from 17 to 22 October 2026. Of these, 3 ended in an infrastructure error and were rerun once, as pre-registered; none failed twice. Message limits were 10 per minute and 2,000 characters each.
The per-run table and the output of the frozen analysis are published with the experiment’s results. They contain no question text, answers or transcripts: BrowseComp-Plus keeps its questions obfuscated to keep them out of training data. The harness code is public.
Limits: one model, one team size (four), two suites and prompted rather than trained norms. All agents share one machine’s throughput, so part of any speed difference is batching. The answer check can miss a correct answer written in an unexpected form; that affects every set-up alike. With 60 pairs, the primary interval is wide: it excludes the registered 10-point gain, but includes anything from a loss of 15.0 points to a gain of 6.7.