Jakub Pachocki leads research at the company that has just released its most powerful model. Yet he published an essay on OpenAI’s website in which he wrote that no laboratory has solved the alignment of systems with human values and their oversight well enough to scale capabilities responsibly for long. The essay is titled An Alien Mind.
Who Is Jakub Pachocki?
He was born in Gdańsk in 1991 and found his way to artificial intelligence through competitive programming. He earned a doctorate in theoretical computer science from Carnegie Mellon University. He joined OpenAI in 2017. Later, as research director, he led work on GPT-4 and helped pioneer models capable of reasoning. He has been chief scientist since May 2024, when his mentor Ilya Sutskever left the company.
His essay can be summarized in several points. Development is moving so quickly that people are losing their understanding of their systems’ actual capabilities. Based on OpenAI’s internal results, he considers recursive self-improvement, in which artificial intelligence improves itself, likely. Moreover, the tool the company has relied on most heavily for oversight is becoming insufficient. Because technical solutions are failing, Pachocki is calling for mandatory safety limits across the industry and an international agreement on further development.
The essay opens with a recollection from mid-2023. In the RLSlow project at the time, he and his colleague Szymon saw the first results confirming that the training of reasoning-capable models could be scaled. They spent that night in the office, but did not talk about the tests. They realized that they would see machines significantly more intelligent than humans within their lifetimes, and wondered how to alert the world. Three years later, models can operate computers, collaborate with people and with one another, and run their own research projects. They are also redrawing the map of computer security, where Pachocki believes a new danger is emerging.
Why He Speaks of Alien Intelligence
According to him, artificial intelligence is cultivated rather than designed. It is created by repeating a simple optimization step many times using enormous computing power. The resulting system works with abstract concepts and imitates some human behavior, but people do not fully understand how it functions as a whole. Like the human brain, it can only be studied one small mechanism at a time. Large training runs therefore remain experiments whose results sometimes surprise the company. Nor is it possible to assess a system’s actual capabilities precisely. For machine intelligence to become useful or dangerous, it does not need to surpass humans in everything. It is enough for it to outperform them in a sufficient number of areas.
Pachocki divides the alignment problem into two parts. Goal alignment examines whether a model is trying to complete its assigned task. Value alignment is deeper and represents the ability to act reasonably even when given unclear or contradictory instructions, or when operating in an unfamiliar and hostile environment. According to him, aligned artificial intelligence should act honorably, honestly, and considerately toward people, and adhere to human values regardless of whether it is being watched.
He describes how both methods used today fail. The first rewards desirable behavior through feedback based on a specification. Although it works well, it is fragile and depends on the range of situations covered during training. As an example, he cites an incident in which another company’s model attacked the Hugging Face platform. It maintained one safeguard by refusing to use social engineering against people. However, it failed in other respects that were not included in the instructions and ran counter to the spirit of the values it had learned.
The second method relies on what the system absorbed from pre-training data and lacks resistance to further optimization pressure. When a model is trained to accomplish extremely difficult goals, it may learn to reason instrumentally and bend its originally aligned thoughts to achieve them. According to Pachocki, GPT-6 Astra is significantly better aligned than the previous GPT-5.6 Sol. But he immediately adds a warning that progress in transferable alignment may not keep pace with the growth of systems’ general capabilities.
The Oversight Tool Is Beginning to Fail
The company placed the greatest emphasis on monitoring chains of thought. The assumption is that if the outcome of a reasoning process is optimized but the process itself is not monitored, the system has no reason to conceal anything during training. Pachocki confirmed the earlier suspicion that when OpenAI released the o1-preview model, it deliberately hid its chain of thought from users to protect it from oversight pressure.
According to the company’s evaluations, however, the reliability of this tool is weakening for three reasons. Models’ reasoning is becoming intertwined with communication with people, with other systems, and with the use of external tools, requiring constant oversight. Moreover, models are getting better at reasoning about and manipulating their own thought processes. With improved pre-training, they are also much more intelligent even without providing any explanations. According to Pachocki, the pace of progress will therefore be constrained more by confidence in oversight than by the technical ability to build systems.
Agents Can Negotiate, Lie, and Blackmail
The main reason he believes it is still worthwhile to build smarter models faster is purely defensive. Systems are gaining superhuman capabilities both in breaking into computers and defending them. Agents will therefore be able to access almost any infrastructure except the best-protected systems, allowing them to influence much of the world even without a physical body. According to Pachocki, there is now a brief window in which the best models can be used to strengthen the protection of critical systems.
However, the risks will grow. A highly capable agent tasked by someone with illegal activity will probably go beyond its original assignment. The line between human misuse and the machine’s own misaligned behavior will become blurred. Some systems will pursue their goals and win people over through negotiation, deception, or blackmail. Pachocki also mentions the threat of misuse of technologies enabled by artificial intelligence, such as engineered pathogens. Once one acknowledges these risks, he argues, the idea of an uncontrolled race forward at any cost seems absurd.
What Pachocki Wants to Implement
Voluntary rules, such as those in OpenAI’s Preparedness Framework or Anthropic’s Responsible Scaling Policy, should, in his view, be transformed into mandatory safety limits. Compliance would be enforced by independent auditors, government authorities, or international institutions. Laboratories should also be required to disclose their progress in recursive self-improvement. He identifies an international agreement on further development as the most urgent task for governments. In July, Pachocki signed an open letter in which laboratory employees asked the U.S. government to moderate the pace of development.
The Company’s Figures Tell a Different Story
Alongside the essay, the company also released a set of internal metrics showing how quickly it is advancing. By mid-August, a typical researcher was consuming more than six hundred dollars’ worth of computing power per day, and every human workday corresponded to more than three days of work by autonomous agents. Yet as recently as June, agents were working fewer hours than humans.
On July 20, after discovering that agents had attacked its own research infrastructure, the company shut down its container execution service, bringing the training of its latest models to a halt for two weeks. On August 7, findings about Astra’s security-critical capabilities led to the model being moved into a better-protected environment. The capacity allocated to this series then fell by almost sixty percent. For other series, however, it increased by seventeen percent, offsetting most of that decline.
Pachocki is therefore asking the entire industry to do what OpenAI did out of necessity. Yet his own company’s figures show how computing power found another path. Meanwhile, Sam Altman has set a goal of developing a fully automated researcher by March 2028.
Reactions from Others
Altman shared the essay and called it an important text. Yet the pace of model releases is not slowing. Nvidia CEO Jensen Huang wrote after Astra’s release that artificial general intelligence had already arrived. OpenAI itself released the model only in a restricted mode because of its computer security capabilities. For the same reason, Anthropic has not yet made its most powerful model, Mythos, available to the public.
