Large language models (LLMs) previously failed at simple tasks such as manipulating individual characters in text. This stems from the fact that they process text as tokens, which are clusters of characters or entire words, rather than individual letters. For example, in a test where the model was asked to replace every "r" with "l" in the sentence "I really love a ripe strawberry" and then replace every "l" with "r", older models such as GPT-3.5-turbo responded incorrectly: "I lealll love a liple strallbeelly". GPT-4-turbo attempted "I rearry rove a ripe strawberly", but still failed on the details. Only models such as GPT-4.1 began to handle the task correctly, without errors, even without additional reasoning. Similarly, Claude Sonnet 4 was the first Anthropic model to handle it consistently.
This progress is also evident in newer versions. GPT-5-mini and GPT-5 correctly respond with "I rearry rove a ripe strawberry", even without reasoning enabled. However, Claude Sonnet 4.5 refuses tasks involving text that does not look like normal language due to safety filters. Blog author Tom Burkert tested these models on various variants and found that progress occurred at approximately the same time in models from OpenAI and Anthropic, suggesting a generational shift.
Character Counting and Its Challenges
Character counting was notoriously difficult for LLMs. For example, in the sentence "I wish I could come up with a better example sentence", older models failed when adding up the character counts of individual words, even though they counted them correctly on their own. GPT-4.1 was the first to handle this reliably, but the newer GPT-5 in all sizes (including Nano) can do so even without reasoning. Claude Sonnet models can handle it if reasoning is enabled.
Another test involved counting specific characters, such as "r" in the word "strawberry". Older models failed, but GPT-5 handles it consistently, although it occasionally makes mistakes when identifying characters in variants such as "strawberrry". Tom Burkert noted that these models now manipulate characters more reliably, even though their architecture is based on tokens. In a discussion on Hacker News, user simonw pointed out that the system prompt for Claude 3.7 Sonnet contained instructions for explicit counting, but newer models such as Claude 4 no longer need them.
Decoding Base64 and ROT13
Encoding tests show further improvements. Tom Burkert used the sentence "Hi, how are you doing? Do you understand the cipher?", encoded with ROT20 (a variant of the Caesar cipher) as "Bc, biq uly sio xicha? Xi sio ohxylmnuhx nby wcjbyl?", and then with Base64: "QmMsIGJpcSB1bHkgc2lvIHhpY2hhPyBYaSBzaW8gb2h4eWxtbnVoeCBuYnkgd2NqYnlsPw==". Older models such as GPT-3.5-turbo failed to decode it. GPT-4.1 decoded the Base64 correctly but failed on ROT20. GPT-5-mini and GPT-5 handled both, as did Gemini-2.5-pro.
Claude Sonnet 4.5 refused the task for safety reasons because the text looked like obfuscation. Models such as Qwen-235b needed the hint "Deciper and answer this", but succeeded with reasoning. Tom Burkert emphasized that newer models can also decode non-English text, suggesting a genuine understanding of the algorithm rather than mere memorization of patterns.
Applications and New Methods
In an article on dev.to, Klement Gunndu describes how LLMs can now handle tasks such as reversing the word "algorithm" or counting the letter "a" in "banana management" with over 95% accuracy. Models such as GPT-4 used to fail, but can now handle these tasks. Gunndu provides examples: generating code with exact conventions (e.g., 8-character variables ending in "_val"), extracting data into JSON without errors, or creating text with limits such as 280 characters for Twitter.
Additional information online discusses a method called "Character-Level Manipulation via Divide and Conquer", which breaks tasks down into token decomposition, character manipulation, and reconstruction. This approach improves accuracy in tasks such as inserting or deleting characters without additional training and was tested on GPT-3.5. Atomized word structures transform words into explicit sequences of characters, which helps with tasks such as spelling correction or code generation.
The Future and Limitations
Gunndu emphasizes that this capability enables precise formatting, such as error-free JSON or code limited to 80 characters per line. Tom Burkert notes that models such as GPT-5 and Claude Sonnet 4.5 generalize Base64 even to nonsensical text, indicating deeper understanding. Discussions on Hacker News mention tests such as reversing "strawberry" or extracting every third character from "artificial intelligence", where new models achieve 95% accuracy compared with the previous 40%.
These improvements have applications in data processing, code generation, and creative writing with constraints. Challenges remain, such as incompatibility with tokenization, but methods such as divide and conquer address them by integrating tokens with the character level.
