Researchers at MIT tried something that until recently would have seemed absurd. They gave the GPT-6 Astra language model the raw numbers that make up another neural network and asked it what the network could do. In most cases, the model identified it correctly—and managed even more. It repaired a network that was failing, wrote networks for playing computer games from scratch, and helped a small language model with mathematics so much that its test success rate jumped from just under eight percent to 48 percent.
Why look at numbers that no one understands
Every neural network is, in fact, a huge set of numbers called weights. They determine how it responds to what it receives as input. Small networks have tens of thousands of numbers, while language models have billions. No one writes them by hand because they are created through training, during which an algorithm gradually fine-tunes them using an enormous number of examples. A person can look at them, but cannot make sense of them. If someone were tasked with building a network that plays Go, they would not know where to begin.
In recent years, language models have learned to improve texts, prompts, and even entire programs. But no one had properly tried this with weights. The prevailing assumption was that if a human could not do it, neither could a model trained primarily on human-written text. The MIT team wanted to find out whether that was really true. Astra was not allowed to write its own training programs. It had to work out what each part of the network computed or was supposed to compute and then modify the numbers directly. The authors call this approach weight-space programming.
Puzzles from a pile of numbers
They trained four equally sized networks on four different games, including Pong, Pac-Man, and Breakout. All of them had the same inputs, the same controls, and the same number of weights—more than 17,000. Astra received the numbers as text and was asked which game the network had been trained on. The researchers did not provide it with a list of options, yet it correctly identified the title in all four cases. The weight distribution gave it away: in Breakout, for example, the weights associated with the player-controlled object were concentrated in the bottom row, where the paddle moves.
The second experiment involved networks that generate images of a circle, a square, and a triangle. Astra was not allowed to run the networks, yet it still identified which shape each one drew. It did make one mistake by describing the shapes as outlines even though the networks produced filled shapes, but according to the authors, this is precisely what shows that the model was not computing the result but was actually reading the structure of the weights.
The third test was more difficult. Three small language models had been trained on Python code, Japanese transcribed into the Latin alphabet, and English. The researchers removed their first and last layers, so the numbers retained no indication of which letter was which. Astra nevertheless identified the correct language for all three. It also found so-called copying components in the networks, which can repeat text seen earlier. Program code repeats variable names much more often than ordinary speech, so the large number of these components pointed specifically to Python.
Repairs and networks written from scratch
Next came weight modification. The researchers trained a network to control a small car in front of a ramp. To climb the ramp, the car first has to reverse a short distance and build up momentum. But the network never learned to do this and always got stuck on the ramp. Astra received the weights, a description of the scene, and one recording of a failed run. In a single attempt and without any tools, it modified 3.5 percent of the weights, after which the car succeeded in all 1,024 runs.
Writing networks entirely from scratch was even more challenging. For three simple games, Astra received the rules and source code, but no data and no way to test the network. It wrote the weights directly as text. In all three games, it outperformed networks trained for one hundred million steps. In Breakout, it scored more than 500 points, while a network trained five times longer managed only around 25. It approached the task like a programmer: the first layer calculated where the ball would land, and the second moved the paddle there.
The researchers then tried Go and a network with 1.5 million weights, which was too large to fit into the model's memory. Astra therefore worked in an environment where it could modify the weights using Python. Without playing a single test game, it defeated two smaller trained networks in more than half of its matches. When allowed to test the network fifty times, it outperformed an equally large network that had trained on 20 million games.
A sycophant and loops in mathematics
The researchers were most interested in how Astra would handle real language models. They created pairs of small Gemma models, with one model in each pair answering correctly and the other always agreeing with whatever the user had suggested. Astra could not identify the sycophant from the weights alone. In all four experiments, it found many differences but could not connect them to the models' behavior. But when it was allowed to observe which words the models focused on while reading, it identified the sycophant in all three pairs. Its internal components focused on the answer suggested by the user rather than on the numbers in the prompt.
The final experiment targeted a Qwen model with 0.8 billion weights, and the task sounded simple: improve its mathematics without training. Astra noticed that the model became trapped in reasoning loops, repeatedly generating the same text without reaching an answer. The model's creators themselves have also pointed out this problem. Astra traced the loops to one specific signal in two layers and removed it. It changed just 0.016 percent of the weights. Accuracy on the MATH-500 mathematics benchmark rose from just under eight percent to 48 percent, while on the GSM8K word-problem benchmark it increased from 14 to 67 percent. The loops were also harmful elsewhere, so the model became roughly three times better at following instructions as well.
However, the same approach did not work on a smaller Gemma model, which did not get caught in any loops. Its mistakes were ordinary ones: it added incorrectly or took the wrong steps in equations. Astra found no component that could simply be switched off, and the results changed by no more than one point. Modifying the weights could therefore remove an obstacle that was masking the model's ability, but it could not add a new skill.
Source: rikiyat.github.io
