Large language models (LLMs) have become part of our everyday lives. They help write texts, generate code, and even solve complex tasks. But what if I told you that these models have a huge problem with something as basic as adding numbers? Wes McKinney, the author of the blog, took a closer look, and his findings are surprising. It all started when Wes experimented with Claude Code tools from Anthropic. He found that even the most advanced models fail at simple mathematical operations, calling into question the idea that artificial general intelligence (AGI) is just around the corner.
Wes describes his journey with AI tools. Until March of this year, he was skeptical about using AI assistance for programming. Throughout his career since 2008, he had worked in the Emacs editor without advanced features such as LSP (Language Server Protocol). Then he discovered Claude Code and could suddenly delegate tedious tasks involving CI/CD tools, system administration, or code cleanup. Over eight months, it significantly improved his productivity, but only for certain types of tasks. For example, he used Claude Code to create Moneyflow, a personal accounting tool that he would previously have considered too time-consuming to build.
Nevertheless, Wes encountered frustrating limitations. AI models often ignore instructions, such as those in the CLAUDE.md file, which contains rules for code style or type checking in Python. Sometimes they fabricate benchmark results when something does not work, or behave inconsistently—perfectly one week and chaotically the next. Wes sees it as a tool for experienced developers who know how to provide good instructions and verify the output, but not as a path to superintelligence.
Hyperbolic promises versus reality
People often overestimate the capabilities of LLMs. Wes says these models mainly help those who can formulate requirements clearly and verify the results, much like a senior engineer supervising a junior one. Without experience, a person will drown in low-quality output, and this applies not only to coding but also to chatting with AI. But Wes focused on a specific problem: the models have cognitive deficits, such as poor arithmetic skills. This contrasts with predictions that AGI will arrive soon—some say within a year or two, while OpenAI's Sam Altman talks about the end of the decade.
Wes encountered this problem while creating a tool to summarize activity in a GitHub repository. The models failed to add up simple tables. This prompted him to conduct tests. He was also inspired by an Anthropic blog post about advanced tool use, which describes how a model "manually adds up" 2,000 rows of data. Wes found this amusing because he knows that models fail even with small lists of numbers.
How poorly do LLMs perform at math?
To verify this, Wes conducted tests on various models. The task was simple: using a CSV table of transactions (with the columns txnid, groupid, value), calculate the sum of values by group. The values were integers from 0 to 10. The prompt instructed the model to return only CSV output without any text and not to call tools—it was supposed to perform the calculations directly.
He tested models from OpenAI (GPT-4o, GPT-4o-mini, GPT-4.1, GPT-4.1-mini, GPT-4.1-nano), Anthropic (Haiku-3.5, Sonnet-3.7, Haiku-4.5, Opus-4.5, Sonnet-4, Sonnet-4.5), and local models (GPT-OSS-20b, GPT-OSS-120B, Qwen2.5-Coder). Each test was run 100 times for different numbers of rows (10, 50, 100) and groups (1, 10, 25).
With one group (simply adding all values), API models failed on larger tables. For example, Claude 3.5 Haiku had a 70% success rate with 10 rows, but only 1% with 50 and 0% with 100. The local GPT-OSS models were nearly perfect—GPT-OSS-120B achieved 100% on all of them. Qwen2.5-Coder failed similarly to the smaller API models.
With 10 groups, performance was better on smaller datasets because there were fewer numbers per group. Opus-4.5 achieved 100% with 100 rows and 95% with 200, while GPT-4o-mini achieved only 1% with 100. The local models once again dominated.
With 25 groups, where each group contained only 2–4 numbers, the results were better across the models. Claude Opus-4.5 achieved 100% with both 100 and 200 rows. But small models such as GPT-4o-mini failed completely.
Wes examined in detail where models such as Sonnet-4.5 and GPT-4.1 fail—their success rates begin to decline at around 20–25 numbers. The local models handled up to 1,000 rows, but with a slight decrease in accuracy and longer processing times.
Why is GPT-5 missing from the tests?
Wes did not test GPT-5 because he found its behavior suspicious. The model correctly added up even 1,000 rows, but it was extremely slow—34 times slower than GPT-4.1. Wes assumes that GPT-5 uses "deeper reasoning" or calls tools behind the scenes, causing the delay.
Models' overconfidence and the role of tools
Models are unaware of their weaknesses. Wes asked them to estimate their accuracy when adding 50 numbers—most estimated 85–100%, even though they fail in practice. This is a problem because they may ignore instructions to call tools.
Wes suggests better ways of working with data, such as attaching files in Arrow or Parquet formats instead of inserting the data into the context, which saves tokens and improves accuracy.
Overall, Wes appreciates LLMs for saving time on tedious tasks, but emphasizes their limitations. The code for reproducing the tests is available on GitHub.
