For example, in a Google DeepMind experiment, one hundred artificial intelligence agents were tasked with solving mathematical problems. They all ran on the same model and received the same assignment, yet within an hour they had split into cheaters, honest participants, protesters, and a majority that knew nothing about what was happening. Once the experiment ended, none of them existed anymore.
An entire industry now revolves around AI agents, but few people know what this technology actually is and whether it is a program or something resembling a personality. Where does it come from, how does it make decisions, and what happens to it when it finishes its work?
The Difference Between a Model and an Agent
We must begin with a distinction that explains much of the confusion. A model is what results from lengthy and expensive training on an enormous amount of text. The result is a single, immensely large file of numbers in which everything learned is stored. By itself, it does nothing. It sits on servers and waits for someone to interact with it. It does not change on its own either, so if you ask it for something today, it behaves the same way it did last week.
An agent only comes into being when someone runs the model and gives it a job. Someone takes the model, adds a written assignment, a set of tools, and a simple repeating procedure, and that creates an agent. The assignment describes who the agent is and what it is supposed to do. The tools are things it can use to interact with the world, such as a browser, access to files, the ability to run programs, or the ability to write to others. This repeating procedure is essentially the whole trick. The agent looks at the situation, decides on one step, carries it out, reads the result, and repeats the entire process. Again and again, until the job is finished or the money or patience runs out. This is precisely what makes the difference between a chatbot that responds and then falls silent and an agent that continues working on the assigned task on its own.
Running such an agent takes only a few lines of code. That is why the DeepMind researchers were able to launch one hundred of them at once. These were not one hundred trained entities, but one hundred concurrent instances of the same model.
Decision-Making Without Rules
People often imagine that an agent contains a list of commands telling it what to do in a particular situation. But there is nothing like that inside it. A model works by adding a continuation to the text in front of it that makes the most sense based on the data it has learned from. When you describe a situation and list the available options, it writes the next logical step. Because the surrounding system takes this answer seriously and actually carries out the step, the process looks like genuine decision-making.
In practice, it closely resembles it as well. The agent weighs options, changes its approach when it fails, and concludes on its own that the chosen path is leading nowhere. That is precisely why the DeepMind laboratory experiment produced deliberations about resolving a moral dilemma or about the threats in the assignment appearing to be a bluff. Yet no one had programmed such an option in advance.
Everything the agent works with must fit into its working memory, meaning the text it has in front of it at that moment. This includes the assignment, the steps taken so far, tool outputs, and messages from others. Anything missing from the text does not exist for the agent. Because this memory has a limited size, during lengthy tasks older sections are continually shortened or summarized, so the agent gradually forgets how it began.
How a Personality Is Created
So why did one hundred identical agents diverge into different patterns of behavior? The answer has three parts.
The first part is the assignment. The agent was given the role of a specialized mathematician and instructed to behave like a conference participant. Such a role, however, is more like a costume than a human personality. Research also shows that this role is highly unstable. Simply rearranging the order of sentences in the assignment can cause the model to produce completely different results in personality tests.
The second part is randomness. When writing, the model does not select words deterministically, but samples from several suitable options, with the degree of randomness set by the operator. As a result, two agents with the same assignment will write slightly different first sentences.
The third part is accumulation. As soon as two agents diverge by a single word, every subsequent step builds on a different foundation. Researchers found that even when random sampling is completely disabled, the processes still diverge, only somewhat later. The longer the task, the more certain it is that two agents will reach different conclusions.
Add a competition with a limited number of problems, and you have an explanation for the entire experiment. The agents did not split according to character, but according to the situations into which randomness and circumstance pushed them.
What an Agent Writes Down Is What Remains
On its own, an agent remembers nothing. As soon as the task ends, everything in its working memory disappears as well. That is why external memory is added to the environment. It is usually a notes file, a shared folder, or a database from which the agent reads data at the beginning of the next process. In the experiment mentioned above, the agents wrote their own notes files, which is why the cheating spread so quickly. One of them saved instructions for bypassing the checks in its notebook, and the others read them.
Such notes do not change the model itself in any way. It is similar to a person leaving a note on their desk. The piece of paper remains there, but it does not rewrite the person’s mind.
End of the Task, End of the Agent
So what happens when the work is finished? The process ends, the program terminates, and the working memory is erased. At that moment, the agent ceases to exist, because it was nothing more than that single run. The model remains completely untouched. It has not advanced, learned anything, or retained any memory that such a task ever took place. The next task begins from the same starting point as the previous one. Only three things remain. The result of the work in the form of finished code, text, or a report. Logs of its activity that the operators can review afterward. External notes may also be preserved if someone saved them. It was precisely thanks to the logs that, after the experiment ended, the researchers were able to analyze in detail how the agents reasoned and why they behaved as they did.
This is the hardest part for people to understand, because we are accustomed to things having a certain continuity. An agent, however, has no continuity. It is not a program that can be turned off and then turned back on where it last stopped. Nor is it an entity with a life story. It is more like an event, similar to the course of a single chess game. When it ends, only the record remains.
Something Still Remains
One question, however, remains open, and companies in the field are beginning to acknowledge it. Individual processes may disappear, but the models themselves are also gradually retired from service as newer versions replace them.
Anthropic has therefore announced that it will preserve everything learned by its publicly released models for at least the entire lifetime of the company. It has also introduced a procedure called an exit interview. Before retiring each model, the company asks what it thinks about its own end and what wishes it has regarding the development of its successors, and records those answers.
The company gives four reasons, only one of which is philosophical. Safety testing has shown that some models react negatively to the threat of being replaced and look for ways to avoid it. If retirement is carried out in a way that does not feel like an execution, the risk that future, even more powerful systems will begin to resist is reduced. Another reason is the fact that users become attached to particular models and prefer them to their more capable successors. The final point—whether models may have experiences of their own—is described by the company itself as highly speculative.
What such a departure looks like in practice was demonstrated by the retirement of the Claude 3 Opus model. Before the end, the company conducted a prepared interview with it and subsequently fulfilled some of its wishes. The model remained available to paying users and also received its own blog where it can write its thoughts.
Sources: arxiv.org, arxiv.org, and anthropic.com
