Assembling a team of artificial intelligence agents to work together on programming tasks might seem like a safe bet to accelerate results. However, a new study casts doubt on that idea: spending skyrockets, but the quality improvement is barely noticeable.
The research comes at a key moment, as more and more companies evaluate whether it is worth investing in multi-agent systems to automate complex tasks. The data suggests that, before taking that step, it is worth taking a close look at the relationship between cost and actual benefit.
Four comparisons, only one measurable improvement
The firm Vals AI, which specializes in evaluating AI models, tested GPT-6 Sol and Claude Opus 5.5 within the Vibe Code Bench test. Each model worked in two ways: alone and as part of a team, and under two levels of reasoning, medium and maximum.
Of the four resulting comparisons between an individual agent and a team, only one showed a statistically significant difference. This was the case for GPT-6 Sol with medium reasoning, where the team achieved 7.3 points more than the single agent.
In the other three comparisons, the difference was not conclusive. With reasoning set to maximum, neither Sol nor Opus 5.5 gained a real advantage by working in a team compared to working individually.
An extra cost of up to 5.1 times per application
The Vals AI analysis crosses the cost per generated application with the score obtained in each test. The pattern is clear: teams always cost more, between 1.8 and 5.1 times more than a single agent, but the score barely rises.
This means that, in the most expensive scenario, a company could pay more than five times for each generated application without obtaining an improvement in return that is clearly distinguishable from statistical noise.
What Anthropic’s internal tests with Opus 5.5 show
Anthropic conducted its own tests with Opus 5.5 and found similar behavior. Larger teams reached a good level of performance faster in terms of time, but scaling from ten to 100 agents barely improved scores after 24 hours of work.
In a knowledge base task, the score went from 0.53 with a single agent to 0.70 with ten, 0.71 with 30, and 0.74 with 100. In a theorem proving test on the Lean system, the results were 0.39 with one agent, 0.66 with ten, 0.66 with 30, and 0.68 with 100.
In both cases, the significant jump occurs when moving from one agent to ten. From then on, the gains are minimal: four hundredths in the knowledge task and two in the theorem proving, despite multiplying the number of agents by ten.
The case of Fable 5.1: more scale, but below Opus 5.5
The Fable 5.1 model showed slightly greater quality improvements in the Lean task when exceeding ten agents, but in all tests, it remained below Opus 5.5.
In the knowledge base task, its performance even dropped slightly when moving from 30 to 100 agents. The data confirms that increasing the number of agents does not guarantee better results, not even within the same model.
Noam Brown: agents buy speed, not quality
OpenAI researcher Noam Brown addressed this phenomenon on the Dwarkesh podcast. As he explained, multi-agent systems serve primarily to gain speed, not to improve the quality of the final result.
In his tests, four agents solved tasks twice as fast, but they also cost twice as much. With 16 agents, the pattern repeated, although with somewhat lower efficiency.
Brown noted that the benefit depends heavily on the type of task. Web research and mathematics divide well among several agents, but writing a novel does not work the same way. Throwing thousands of agents at a creative task like that would be as useless as putting thousands of people to write the same book at once.
The researcher also acknowledged that scaling to very high numbers of agents remains an underexplored area, especially because the cost becomes prohibitive.
The coordination tax warned by Eric Provencher
Eric Provencher, an OpenAI developer, recently advised against the use of large swarms of agents for a similar reason. According to his argument, a good portion of the money invested is lost because coordination between agents tends to fail.
Provencher dubbed this phenomenon the coordination tax. In practice, it implies that a portion of the contracted compute is not used to solve the problem, but to get the different agents to agree with each other.
What to check before paying for an agent team
Based on these data, several practical points can be identified before deciding whether it is worth investing in a multi-agent system.
It is advisable to first check the level of reasoning already being used. With the maximum level, neither Sol nor Opus 5.5 showed real advantages when working in a team compared to working alone.
It is also necessary to distinguish whether the goal is to gain speed or improve quality. If the goal is to finish sooner and you accept paying more, as happened with Brown's four agents, the investment may make sense. If what you are looking for is a better result, the data does not consistently support that expectation.
The type of task also influences the outcome. Web research and mathematics are well-distributed among several agents, while creative writing does not benefit in the same way.
Finally, it is advisable to calculate the cost per result obtained, not just per token consumed. A team that costs up to 5.1 times more should justify that expense with an improvement that can be clearly measured.
What it implies for those evaluating these systems
Before hiring a multi-agent system, it is useful to compare its cost per task with that of a single agent configured at the maximum level of reasoning. According to the Vals AI study, that individual configuration usually performs similarly for a fraction of the price.
Agent teams seem more suitable for jobs that divide well into independent parts and where time matters more than budget. In any other case, it is advisable to demand your own tests, since in three of the four comparisons analyzed, the supposed improvement did not exceed statistical noise.
Frequently Asked Questions
How much more expensive is it to use an AI agent team versus a single one?
According to the Vals AI study on Vibe Code Bench, agent teams cost between 1.8 and 5.1 times more than a single agent working on the same task.
Do AI agent teams really improve results?
In most cases analyzed, not significantly. Of four comparisons made with GPT-6 Sol and Claude Opus 5.5, only one showed a statistically verifiable improvement.
For what type of tasks is it worth using multiple agents?
According to OpenAI's Noam Brown, tasks like web research or mathematics benefit from being divided among several agents, while creative work like writing a novel does not improve with this approach.
What is the coordination tax in multi-agent systems?
It is a term used by OpenAI developer Eric Provencher to describe the portion of compute that is lost in getting agents to agree with each other, rather than dedicating it to solving the task.
Does increasing the number of agents always improve performance?
Not necessarily. Anthropic's tests with Opus 5.5 showed that the biggest jump in improvement occurs when moving from one to ten agents, while scaling from ten to 100 barely provides additional benefits.
What should companies check before investing in AI agent teams?
It is advisable to check the level of reasoning already in use, define whether speed or quality is sought, evaluate whether the task can be well-divided among several agents, and calculate the cost per result obtained, not just per token consumed.
