https://arxiv.org/abs/2509.23537

Massgen makes different AIs work together as a team, debating and voting on answers. This collaborative approach overcomes individual weaknesses, achieving smarter results than any single AI could alone

Introduction

For years, the story of AI progress was simple: bigger models, better results. But even today’s frontier models — billions of parameters, trained on oceans of data — show uneven strengths. One dominates logic, another instruction-following, and another narrative reasoning.

So we asked a different question:

What if multiple language models could deliberate together — not just once, but across multiple rounds of debate, reflection, and consensus?

This is the idea of multi-turn multi-agent orchestration: treating models not as isolated geniuses, but as participants in a collective.

The results are striking: groups of models coordinating through structured debate rival or surpass the strongest individuals. What emerges is not just better performance, but a glimpse of synthetic social intelligence.

From Soloists to Ensembles

Our orchestration framework gives each model a clear role in every round:

Propose — generate a candidate answer.
Vote — evaluate peers’ answers and cast a vote.

Agents act asynchronously. Whenever a new candidate appears, voting restarts — like a panel discussion interrupted by fresh evidence. Once all votes are cast, the majority answer wins, and the “winning” agent synthesizes all reasoning into a unified final response.

This design prevents premature convergence, ensures diversity of thought, and enables peer correction before consensus is reached.

Four Minds Enter the Arena

We tested orchestration with four of the strongest models available:↳

Gemini 2.5 Pro
GPT-5
Grok 4
Claude Sonnet 4

They debated across three challenging benchmarks:

GPQA-Diamond — graduate-level expert reasoning.
IFEval — rigorous instruction-following tests.
MuSR — narrative reasoning and multi-step logic.

The ensemble matches or exceeds the strongest soloist.

Model	GPQA-Diamond	IFEval	MuSR	Avg
Grok 4	85.4	84.7	67.6	79.2
GPT-5	84.8	87.4	69.2	80.5
Gemini 2.5 Pro	85.9	66.0	69.6	73.8
Claude Sonnet 4	68.2	63.6	62.8	64.9
Orchestration	87.4	88.0	68.3	81.2

Collective Accuracy Beyond the Ceiling

The ensemble not only matched but sometimes exceeded the best single model:

87.4 % on GPQA-Diamond
88.0 % on IFEval
68.3 % on MuSR

Even more telling: in 95 % of failures, at least one model had already given the correct answer. The truth was almost always present — but not always recognized.

This suggests orchestration is still in its infancy. With better coordination, collective accuracy could push well beyond any individual ceiling.

Effect of coordination strategies on GPQA-Diamond. Bars show percentages under three settings: Default (Anonymous + Hidden Tally), Identified Voting, and Visible Tally. Left: Self-voting Rate, the percentage of votes an agent cast for its own answer. Middle: First-voted Selected Rate, the percentage of tasks where the answer that received the first vote became the final consensus. In the first two plots, the rightmost group "All agents" aggregates across agents. Right: Consensus Tie Rate, the percentage of tasks with no majority. The hatched bar ("≥2 Self-voters") marks the subset of tie cases where at least two agents voted for themselves. Models: Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4.

We explored how coordination rules shape group behavior:

Identity Disclosure — When agents knew which model produced which answer, self-voting surged. Dominant models like GPT-5 reinforced themselves, and tie rates nearly doubled.
Vote Visibility — When votes were visible, herding emerged. Early votes skewed later ones, accelerating convergence but sometimes onto the wrong answer.

These behaviors mirror human dynamics: ego, peer pressure, and groupthink. Even in silicon, collectives inherit social psychology.

How Machines Debate: Case Studies

Self-Correction — In one trial, a model miscalculated a cosmological distance. After peers pointed out inconsistencies, it revised its reasoning, pulling the group to the correct answer.
Persuasion Failure — In another, a confident but wrong agent swayed the majority, proving that consensus can amplify overconfidence as well as insight.

These cases show both sides of collective reasoning: learning through disagreement, failing through persuasion.

From Models to Societies

Multi-turn orchestration suggests the next leap in performance may not be architectural, but organizational. The implications:

Adaptive Coordination — Learning when to reveal identities, when to keep votes anonymous, how to reduce herding.
Meta-Learning Debate — Using reinforcement signals to tune the orchestration strategy itself.
Collective Benchmarks — Evaluating not just correctness, but deliberation quality, stability, and resilience.

What emerges is not just stronger benchmarks but the beginnings of synthetic societies: machines that deliberate, dissent, and decide together.

Conclusion: The Strongest Mind May Be Many

No single LLM dominates every task. But when orchestrated, diverse models combine their strengths, correct one another, and — sometimes — transcend their parts.

The lesson is clear: The age of scaling parameters is giving way to the age of scaling minds.