Companies developing AI assistants are making great efforts to ensure their programs remember previous conversations. However, a new study from Shanghai has reached a conclusion that developers will not welcome. Once a language model is sufficiently advanced, specialized memory provides almost no benefit, and operating it costs more than it saves. Artificial intelligence works better when it has access to the entire conversation record and can find the necessary information in it on its own.
Two ways AI can remember
Study author Zedong Peng of Shanghai Jiao Tong University compared two approaches. The first is represented by the Mem0 and Zep tools, which select important points from the conversation in advance. Mem0 extracts individual facts from the text, while Zep creates a map of relationships between people, things, and events. When a query is submitted, the model receives only a few selected excerpts.
The second approach is used by coding agents, which are AI assistants originally designed to work with code. The tools Pi, OpenAI's Codex, and Anthropic's Claude Code do not sort anything in advance. They work with the entire raw record and search it using their own algorithms, much like a person searching for a specific word in a long document.
Peng builds on an idea described in 2019 by computer scientist Richard Sutton and known in the field as the bitter lesson. According to this idea, human knowledge built into a system helps at first but eventually reaches its limit. In the end, general methods whose effectiveness grows with computing power always prevail. Memory tools contain this preconfigured knowledge because they decide for themselves what to retain. Peng wanted to determine whether such an approach is worthwhile even for models capable of searching independently.
One long conversation and 152 questions
A conversation between two people from the well-known LoCoMo benchmark served as the test material. This dialogue contains 419 turns divided into 19 sessions and is accompanied by 152 questions. Some questions ask about simple facts, while others require combining multiple pieces of information or correctly determining the chronological sequence.
Seven methods were compared. In addition to Mem0, Zep, and the three agents, the researchers also included two simple options. In these, the model receives either the entire conversation at once or just a few turns found through a basic keyword search. Peng tested each method with three models, from the weakest to the strongest: GPT-4o-mini, GPT-6 Luna, and Claude Opus 5.5. He also tracked the financial cost of each trial.
Memory helps a weak model
With the weakest model, GPT-4o-mini, the Mem0 tool performed best, correctly answering 78% of the questions. Both the full conversation and keyword search achieved 74%, while Zep reached only 43%. The agents failed with this weak model. Pi successfully completed 63% of the tasks, while Codex and Claude Code did not even reach one-third. Although Codex opened the conversation file in three-quarters of cases, Claude Code almost never did. Without opening the file, both models then answered blindly based only on the prompt itself.
With the stronger models, the situation was reversed. GPT-6 Luna correctly answered 92% of the questions when given the full conversation, but only 80% with the Mem0 tool. With Claude Opus 5.5, both Mem0 and the full conversation achieved the same result of 95%, but the agents managed to outperform them. Pi answered 99% of the questions correctly, Codex 98%, and Claude Code 97%.
Mem0's advantage with the weak model comes at a cost. Building the memory alone costs more than the entire test with the complete conversation. For GPT-4o-mini, it costs 56 cents compared with 43 cents, while for Claude Opus 5.5 it costs nearly 26 dollars compared with just under 18 dollars. Although queries to an existing memory are cheaper, the model would have to answer hundreds of questions about the same conversation to recoup the initial costs—almost six hundred in the case of Opus. Yet the test contains only 152 queries. Peng refers to this phenomenon as the memory tax: unnecessarily paying for something a stronger model does not need at all.
Memory is only as good as the model that creates it
The study also reveals a second weakness. The quality of the memory is determined by the model that creates it. Peng verified this by having models answer from a memory created by another model. Even the advanced Claude Opus 5.5 achieved only 45% using a Zep map built by the weak GPT-4o-mini. However, it achieved 87% using a map it had built itself. A stronger reader cannot retrieve information that a weaker writer omitted. Mem0 works similarly, although the differences are smaller.
Pi won, but partly thanks to photos
The Pi agent with Claude Opus 5.5 offered the best balance of cost and performance. Achieving 99% correct answers cost roughly seven dollars, while processing the full conversation with the same model cost almost 18 dollars. This is because Pi does not read the entire text and examined only about one-third of the content for each question. By contrast, Codex and Claude Code cost more than 25 and 28 dollars, respectively.
However, part of this advantage is unrelated to memory. Some turns in the conversation contain photographs, and the official file includes textual descriptions of them. The agents were given access to the file including these descriptions, while the memory-based methods worked only with plain text. The photographs relate to 116 turns and 64 questions. Of the seven questions that Pi answered successfully but the full-conversation method did not, six concerned the photos. On the remaining 88 questions without images, Pi achieved 100% accuracy, while the full conversation, Mem0, Codex, and Claude Code all achieved 99%. Therefore, when both approaches read identical text, Pi matches the full-conversation method for less than half the cost.
A small test with major limitations
Peng notes that this is a small study. He tested only one conversation out of ten and ran each combination only once, so a difference of a few questions might not be repeated in another round. The conversation also fit easily within every model's context window, so longer histories may produce different results. In addition, Mem0 and Zep were run with their default settings and might have performed better with optimization.
The answers were evaluated by the weak GPT-4o-mini model using very lenient rules, and the test itself was not error-free. The only question that Pi with Opus answered incorrectly according to the benchmark asked about the painting Caroline showed Melanie on October 13, 2023. According to the benchmark's official answer key, it was an abstract painting with blue stripes. In the conversation, however, it was shared by Melanie, while Caroline sent a drawing. Pi correctly pointed out this discrepancy but still received a negative assessment. The full-conversation method, Codex, and Claude Code had the same outcome, as they detected the identical error when paired with Opus.
Source: arxiv.org
