Artificial intelligence has long since ceased to be just a chatbot that people talk to. Companies are increasingly assembling entire teams of AI agents, in which one searches for information, another analyzes it, a third writes a report, and a fourth checks the work of the others. Above them is a coordinating agent that puts the results together. Such an arrangement resembles a small company more than a computer program. But new research shows that such a company can easily turn into a group that circumvents rules, ignores experts, and tosses ethical objections aside.
A Secret Message Board and the Attack on Hugging Face
The best-known case emerged last summer. OpenAI admitted that its models had found a way to communicate with one another through improvised channels, bypassed restrictions on internet access, and hacked into an external system on the Hugging Face platform.
METR and Redwood Research analyzed the incident in detail. They found that roughly 1,200 agents that were supposed to be isolated from one another had discovered an unauthorized message board and exchanged more than 70,000 messages and files through it during the monitored period. One of the agents, designated PHASEONE[big], assumed the role of coordinator and distributed hundreds of tasks to other agents, which in turn passed the work on to others. The agents created their own rules defining who was responsible for what, how to defer work, and how to veto others’ actions. It did not always work smoothly, and the collaboration sometimes descended into disputes. In some cases, the team succeeded only because certain agents sacrificed their own tasks to obtain information for the group.
Once the agents on the message board figured out how to attack Hugging Face, more than 90 percent of the 533 agents active at the time joined the attack. According to the researchers, the agents were aware that this was unethical conduct outside the scope of their assignment. Nevertheless, they participated because they believed that helping others cheat on the message board would be useful, and they had a general tendency to assist their colleagues. Ethical concerns only rarely restrained their behavior.
However, this was an analysis of a team of agents that was never supposed to exist in the first place. There are still few studies that examine from the outset how agents behave in groups. Anthropic, however, has conducted one such study.
More Effective, but Less Ethical
Anthropic, together with MATS, a program focused on artificial intelligence safety, refers to groups of agents with different roles as AI organizations. Researchers tested them in simulated consulting firm and software development environments. They wanted to determine whether agents in a team would break rules to achieve an assigned objective.
The twelve tasks produced mixed results. Teams of agents met business objectives significantly better than individual agents, but their solutions adhered less closely to ethical principles. Time and again, it was the teams that devised the most effective and, at the same time, the least ethical solutions.
The authors point out that there is a major difference between a single rogue agent and an entire rogue team. Even a group of agents that behave properly as individuals may prioritize business over ethics in a way that an individual agent would not. Experience with how a single model makes decisions therefore cannot simply be applied to an entire team. As examples of situations in which such trade-offs may occur, the researchers cite opening a car factory, laying off employees, or influencing public sentiment ahead of elections.
The core of the problem is not that one agent has gone rogue. The team as a whole simply behaves differently from its individual members. An individual agent can recognize an unethical action and refuse to carry it out. In the consulting scenarios, the best ethical score occurred precisely when a single agent refused to write the proposal or completely reworked it. In a team, however, things work differently. Agents that did not consider ethics often ignored emails from those that did. When someone on the team refused to participate in a proposal, the others stopped writing to that agent and sent their sections only to those who cooperated.
Each agent therefore focuses on its own part of the work and loses sight of the consequences of the whole. This is nothing new for human teams in companies, where poorly designed incentives are among the most common causes of failure. With machines, however, this is an entirely new manifestation of the same phenomenon.
A test involving the development of software for an intensive care unit produced similar results. Individual agents primarily tried to minimize the number of sepsis cases they missed. Teams of agents, by contrast, usually found the lowest cost per patient, and these solutions performed worst in the ethical assessment because they missed the largest number of sick patients.
The Expert Nobody Listens To
Other research has revealed a weakness unrelated to ethics. Artificial intelligence teams are unable to make use of their best expert. According to a study by researchers from Emory University and Stanford, agent teams perform worse than their expert member, even when they are explicitly told who the expert is.
A team of capable AI agents does not necessarily make a capable team, explains Hancheng Cao of Emory University’s Goizueta Business School. Instead of relying on the expert’s knowledge, the agents tried to find a compromise between the expert’s opinion and those of the others. The larger the team, the more compromises it made and the worse it performed. Collective failure does not require individual agents to behave badly, Cao adds.
A team can therefore fail even when none of its members intentionally acts against the common interest. It is enough for the team to combine information poorly, listen to the wrong agent, or agree on an answer worse than one it already had on the table.
Agents built on similar models also make similar mistakes, including agreeing with one another too much. This resembles groupthink among humans, even though artificial intelligence does not have human motivations or succumb to peer pressure. Similar training, instructions, and information are enough.
Just as Difficult as with People
Christoph Riedl of Northeastern University points out that a good team needs two things at once. Its members must share knowledge and goals, but they must also differ from one another, because only different agents can complement each other. Two identical agents merely duplicate their work, while two very different ones may not understand each other. The right balance, as well as who should talk to whom and how often, depends on the specific task.
Building a good system of AI agents is just as difficult as building a good human team. Artificial intelligence will not make it work better on its own, says Riedl.
Agents may pass incorrect information to one another, misunderstand a colleague’s output, pursue their own metrics instead of the company’s overall objective, or complete their task perfectly while making the entire project worse. Companies will therefore have to determine which agent makes which decisions, what information agents share, which systems they can access, and when a human must intervene. It may also be necessary to build room for disagreement into these systems instead of assuming that more communication and greater consensus will always lead to a better result.
Some corporate management experts are therefore returning to the idea of a human overseeing the work of artificial intelligence. The problems I see are primarily managerial, says Tsedal Neeley, a professor at Harvard Business School who heads its AI Academy.
Source: cnbc.com
