The largest language models can process more text at once than Tolkien’s The Lord of the Rings or Tolstoy’s War and Peace. GPT 5.6 Sol can handle more than a million tokens, or roughly 790,000 English words. But twelve Nvidia researchers asked an uncomfortable question: do models really need such enormous memory? In most cases, their answer is no. A study with the telling title You don’t need long context shows that AI agents can handle even demanding tasks with significantly shorter context, often performing just as well or even better.
The longer the memory, the slower the model
Every new token a model generates must take into account everything that came before it. The more text accumulates in a conversation, the more computation and memory each additional word requires. This is particularly noticeable with AI agents. When programming, an agent searches repositories, edits files, and runs tests, and each such step is added to the next request sent to the model. The history keeps growing, even though most of it has long since ceased to be useful.
The researchers measured this on three open-weight models running on Nvidia H100 graphics cards. When they reduced the context window from 256,000 to 16,000 tokens, the GLM 5.2 model generated roughly 3.6 times as many tokens in the same amount of time, Kimi K2.6 generated about 5.6 times as many, and Qwen 3 Coder Next generated seven times as many. On the smaller DGX Spark device, generation time with long context was roughly double that with short context for several models. The authors note that ordinary users do not see these differences because providers charge a fixed price per million tokens regardless of conversation length.
An agent does not need to remember everything
The team tested 13 benchmarks, including SWE-bench Verified with real bugs from GitHub, Terminal-Bench, and LongBench v2. The main experiments ran on the Claude Opus 4.5 and GPT 5.6 Sol models. The baseline version remembered the entire conversation. When the modified version reached a set limit, such as 12,000 tokens, it had the model briefly summarize the conversation and started a new one containing only the original assignment and this summary.
The results were surprising. On SWE-bench Verified, Claude Opus 4.5 solved 70 percent of tasks with the full history and nearly 75 percent with regular compression under the best configuration. GPT 5.6 Sol improved from 84.4 to 88.8 percent. Across long agentic tasks, accuracy was generally only a few percentage points above or below the baseline, while estimated costs fell by roughly two to nine times.
Much depends on what the summary looks like. The researchers found that a conventional summary often preserves the wrong information. It recounts notable events from the agent’s history even when they are no longer relevant. Something resembling a handover briefing worked much better. In it, the model records the original objective, verified facts, important files and results, changes made, dead ends not worth revisiting, remaining work, and the immediate next step.
It is enough to know where to look
For tasks involving enormous documents, the authors chose a different approach. Instead of inserting the full texts into the prompt, they saved them as files. The agent searched them using standard commands such as grep and brought only the passages it actually needed into its memory. This method matched or outperformed the conventional approach in nearly all seven tests with long inputs. In the needle-in-a-haystack test, both models achieved a 100 percent success rate. Claude Opus 4.5 improved from 65 to 84 percent on the AA-LCR task, which requires reasoning across several real-world documents. The agents also often completed the work without needing to compress their memory even once. The main benefit therefore came not from summarizing what had been read, but from intelligently selecting what to read in the first place.
The team then tried an extreme approach. After every individual action, the agent rewrote the description of the task’s current state and discarded the previous conversation. The next step therefore began with only the assignment and this up-to-date description. Even so, both models nearly matched the full-history version on SWE-bench Verified. Meanwhile, the median length of the longest conversation fell to just 7,000 to 9,000 tokens, and the maximum dropped from a range of 130,000 to 200,000 to between 15,000 and 37,000.
From this, the researchers derived the idea of so-called semantic state models. Instead of passing unreadable numbers within a neural network between steps, these models would learn to pass along a comprehensible textual description of the task’s state that a human could also read. According to the authors, there is no reason why such models could not achieve the same results as those that retain the entire history.
Not every model can handle it
However, the study has its pitfalls. Models respond very differently to memory compression. On SWE-bench Verified, Qwen 3 Coder Next fell from 53 percent to less than 4 percent with a limit of 16,000 tokens. Its verbose outputs were to blame, as they left no room for actual work between summaries. In contrast, Qwen 3.6 27B and MiniMax M2.7 performed much better and lost very little. According to the authors, developers should therefore test everything on their own tasks and with different limits instead of blindly relying on a single configuration.
An unpleasant surprise may also await when it comes to spending on cloud models. Providers offer a discount when consecutive requests share the same beginning and the model can reuse earlier computations. However, memory compression changes that beginning and eliminates the discount. At standard per-token prices, the cost of GPT 5.6 Sol on SWE-bench Verified therefore paradoxically rose from roughly $168 to $203, even though the model processed shorter conversations. The savings calculated in the study assume that providers would charge less for shorter context, which is not yet common practice.
The authors also acknowledge another limitation. A summary always loses something, and it is not worth relying on one for tasks where every detail matters, such as precise records or audit trails. They therefore recommend retaining the original data separately so that the agent can retrieve it at any time. They ultimately support their argument by invoking Alan Turing, who as early as the 1930s worked from the assumption that anything computable can be computed even when one looks at only a limited portion of the input and keeps only a finite state in mind.
Sources: arxiv.org and cryptobriefing.com



