How Much Have AI Models Improved at Arranging and Understanding Letters?

How Much Have AI Models Improved at Arranging and Understanding Letters?

Ondřej Barták
Ondřej Barták
Entrepreneur and Programmer
16. 10. 2025
4 minutes reading · 12 views
How Much Have AI Models Improved at Arranging and Understanding Letters?

Large language models (LLMs) previously failed at simple tasks such as manipulating individual characters in text. This stems from the fact that they process text as tokens, which are clusters of characters or entire words, rather than individual letters. For example, in a test where the model was asked to replace every "r" with "l" in the sentence "I really love a ripe strawberry" and then replace every "l" with "r", older models such as GPT-3.5-turbo responded incorrectly: "I lealll love a liple strallbeelly". GPT-4-turbo attempted "I rearry rove a ripe strawberly", but still failed on the details. Only models such as GPT-4.1 began to handle the task correctly, without errors, even without additional reasoning. Similarly, Claude Sonnet 4 was the first Anthropic model to handle it consistently.

This progress is also evident in newer versions. GPT-5-mini and GPT-5 correctly respond with "I rearry rove a ripe strawberry", even without reasoning enabled. However, Claude Sonnet 4.5 refuses tasks involving text that does not look like normal language due to safety filters. Blog author Tom Burkert tested these models on various variants and found that progress occurred at approximately the same time in models from OpenAI and Anthropic, suggesting a generational shift.

Character Counting and Its Challenges

Character counting was notoriously difficult for LLMs. For example, in the sentence "I wish I could come up with a better example sentence", older models failed when adding up the character counts of individual words, even though they counted them correctly on their own. GPT-4.1 was the first to handle this reliably, but the newer GPT-5 in all sizes (including Nano) can do so even without reasoning. Claude Sonnet models can handle it if reasoning is enabled.

Another test involved counting specific characters, such as "r" in the word "strawberry". Older models failed, but GPT-5 handles it consistently, although it occasionally makes mistakes when identifying characters in variants such as "strawberrry". Tom Burkert noted that these models now manipulate characters more reliably, even though their architecture is based on tokens. In a discussion on Hacker News, user simonw pointed out that the system prompt for Claude 3.7 Sonnet contained instructions for explicit counting, but newer models such as Claude 4 no longer need them.

Decoding Base64 and ROT13

Encoding tests show further improvements. Tom Burkert used the sentence "Hi, how are you doing? Do you understand the cipher?", encoded with ROT20 (a variant of the Caesar cipher) as "Bc, biq uly sio xicha? Xi sio ohxylmnuhx nby wcjbyl?", and then with Base64: "QmMsIGJpcSB1bHkgc2lvIHhpY2hhPyBYaSBzaW8gb2h4eWxtbnVoeCBuYnkgd2NqYnlsPw==". Older models such as GPT-3.5-turbo failed to decode it. GPT-4.1 decoded the Base64 correctly but failed on ROT20. GPT-5-mini and GPT-5 handled both, as did Gemini-2.5-pro.

Claude Sonnet 4.5 refused the task for safety reasons because the text looked like obfuscation. Models such as Qwen-235b needed the hint "Deciper and answer this", but succeeded with reasoning. Tom Burkert emphasized that newer models can also decode non-English text, suggesting a genuine understanding of the algorithm rather than mere memorization of patterns.

Applications and New Methods

In an article on dev.to, Klement Gunndu describes how LLMs can now handle tasks such as reversing the word "algorithm" or counting the letter "a" in "banana management" with over 95% accuracy. Models such as GPT-4 used to fail, but can now handle these tasks. Gunndu provides examples: generating code with exact conventions (e.g., 8-character variables ending in "_val"), extracting data into JSON without errors, or creating text with limits such as 280 characters for Twitter.

Additional information online discusses a method called "Character-Level Manipulation via Divide and Conquer", which breaks tasks down into token decomposition, character manipulation, and reconstruction. This approach improves accuracy in tasks such as inserting or deleting characters without additional training and was tested on GPT-3.5. Atomized word structures transform words into explicit sequences of characters, which helps with tasks such as spelling correction or code generation.

The Future and Limitations

Gunndu emphasizes that this capability enables precise formatting, such as error-free JSON or code limited to 80 characters per line. Tom Burkert notes that models such as GPT-5 and Claude Sonnet 4.5 generalize Base64 even to nonsensical text, indicating deeper understanding. Discussions on Hacker News mention tests such as reversing "strawberry" or extracting every third character from "artificial intelligence", where new models achieve 95% accuracy compared with the previous 40%.

These improvements have applications in data processing, code generation, and creative writing with constraints. Challenges remain, such as incompatibility with tokenization, but methods such as divide and conquer address them by integrating tokens with the character level.

Advertisement

Content created with help from UpTier.

SEO and GEO on autopilot. UpTier’s multi-agent systems write and optimize content for search engines and AI answers.

Discover UpTier ↗

Category:AI
Did you enjoy this article?
Discover more interesting posts on our blog
Back to blog

Related posts

Google makes Tasks free for Gemini 4 Hydrogen usersGoogle makes Tasks free for Gemini 4 Hydrogen users
Tasks is becoming available for free to hundreds of millions of Gemini 4 Hydrogen users. Google says it can call businesses, book appointments and complete purchases on their behalf.
1 min read
18. 11. 2026
OpenAI introduces GPT-6.2 Astra and Sol for professional and everyday workOpenAI introduces GPT-6.2 Astra and Sol for professional and everyday work
OpenAI has introduced two models: Astra for complex professional tasks and Sol for everyday work. Both are available in ChatGPT Work, Codex and through the API.
3 min read
17. 11. 2026
Tencent releases Hy4 Flash with image support and a context window of up to 1 million tokensTencent releases Hy4 Flash with image support and a context window of up to 1 million tokens
The lighter Hy4 Flash model targets chat, documents and routine code edits. It accepts text and images, supports tool use and, according to Tencent, matched Claude Haiku 5.5 in tests of everyday tasks.
1 min read
16. 11. 2026
Přihlaste se k odběru našeho newsletteru
Zůstaňte informováni o nejnovějších příspěvcích, exkluzivních nabídkách, a aktualizacích.
CodedTrip

Operated by CodedTrip LLC, USA.

YouTube
TikTok