A model trained on simple examples first memorizes everything. Then it stops making progress. It looks as though it has become stuck. And then, after further training, something changes. Accuracy on unseen data jumps from random guessing to nearly perfect knowledge. This phenomenon was named grokking. It was first described by OpenAI researchers in 2022 in a paper published on arXiv. Ever since, it has kept many people who study how neural networks actually work awake at night.
What grokking is and why it surprised us
Neural networks normally behave predictably. You train them, the error rate falls, and performance improves. Then it levels off. This principle underpins all of machine learning. Grokking changes this pattern.
OpenAI researchers trained models on small datasets consisting of mathematical operations. Specifically, division modulo 97. The model quickly memorized the correct answers to the training examples, with accuracy approaching 100 percent. But on data it had never seen before, its answers were practically random. A typical case of overfitting—nothing that would surprise the researchers.
The surprise came later. When the researchers allowed the model to train significantly longer than was necessary to memorize the data, accuracy on the test set suddenly began to rise. Not gradually, but in a leap. In one experiment, this happened only after one million training steps, even though the model had managed to memorize the data in fewer than a thousand steps. A thousandfold difference. The model first overfit and then generalized on its own. Without any external intervention.
How the research was conducted and what it revealed
The original study used simple mathematical operations in the form a ◦ b = c, where a, b, and c are discrete symbols. The researchers tested different types of operations, datasets of various sizes, and different model architectures.
One finding was particularly striking. The smaller the dataset, the more training steps were needed for grokking to occur. Small datasets simply require much more computational work before the model transitions from memorization to genuine understanding. Most importantly, the phenomenon appeared across different models and optimization methods. It was not a coincidence or an artifact of one particular architecture.
Neel Nanda, a researcher at Anthropic, decided to investigate grokking directly inside a trained model. He took a simple transformer model and taught it addition modulo 113. The model followed the familiar pattern: memorization, stagnation, and then a leap to nearly perfect accuracy on new data.
Nanda then literally took the model apart. He used mechanistic interpretability to do so. This is an approach that attempts to reverse-engineer algorithms directly from the weights of trained networks. And no one expected what Nanda found there.
The model had developed its own algorithm based on the discrete Fourier transform and trigonometric identities. It converted numbers into frequencies, combined them using rotations around a circle, and then read the answer from the result. It was as if it had independently derived polar coordinates and number theory. No one had taught it this. No one had told it what the Fourier transform was, nor had anyone programmed it to work with circles. The model derived all of it by itself, purely from examples.
What happens inside when we see nothing on the outside
One of the most interesting findings from Nanda's analysis concerns what happens during the seemingly empty phase between memorization and grokking. Outwardly, the model stagnates. Accuracy on the test data remains low. It looks as though nothing is happening at all, but internally, the situation is different.
During this phase, the model builds a circuit that genuinely generalizes. At the same time, however, the old circuit that merely memorizes examples is still running. It adds noise and prevents the generalizing circuit from fully expressing itself. Grokking occurs when the generalizing circuit becomes strong enough for the model to remove the memorization circuit on its own. It clears it away. The completed algorithm is then revealed underneath. It is as though two solutions were competing for space within the model. One must give way for the other to win.
Nanda points out that this is directly related to phase transitions—that is, sudden leaps in model capabilities during training. These also occur in large models trained on real-world data. For example, as transformers scale, they suddenly acquire the ability to learn from contextual patterns. This ability does not emerge gradually, but appears in a leap when the model creates a specific circuit known as induction heads.
The structure of this transition is very similar to grokking. The model secretly builds a capability that does not manifest outwardly until it is mature/complete. Then it switches over. This suggests that much of what we perceive as mysterious sudden behavior in large models may have exactly the same mechanism as grokking in simple arithmetic.
A glimpse inside the black box
For a long time, large models were assumed to be inherently opaque. We know what they produce, but not why. Mechanistic interpretability challenges this assumption. Nanda's analysis of grokking is one of the few cases in which researchers have managed to completely reverse-engineer an algorithm directly from a network's weights—and to prove that this very algorithm is responsible for the model's performance.
A model trained solely on examples independently derived mathematics that took humans centuries to develop. And we discovered this simply by looking inside. How many other mechanisms are still waiting there?
Source: alignmentforum.org
