Ilya Sutskever is a co-founder of OpenAI, where he served as chief scientist until 2023. He now leads Safe Superintelligence Inc. (SSI), which focuses on developing safe superintelligence. In a recent podcast with Dwarkesh Patel, he discussed key challenges in artificial intelligence (AI), such as model generalization and the transition from scaling to intensive research.
Problems with AI Model Generalization
Sutskever describes how current AI models, such as those trained using pre-training and reinforcement learning (RL), fail to generalize. For example, a model may excel at competitive programming, but repeatedly introduce new problems when fixing bugs in code. This resembles a student who trains for 10,000 hours on specific tasks but lacks “that special something”—an intuitive understanding that enables broader application of knowledge. Humans generalize better because their learning is more robust and less dependent on vast amounts of data. Sutskever noted that children learn to drive a car after 10 hours of practice despite having limited experience with visual perception, suggesting an evolutionary advantage in learning.
AI models suffer from uneven performance. Although they achieve high scores in tests (evals), their economic impact lags behind. Sutskever suggests that RL training makes models too narrowly focused and driven by evaluation, limiting their ability to solve real-world problems. By contrast, humans have built-in emotions that function as a value function, helping them learn from mistakes more quickly without having to wait for the final outcome.
The Transition from Scaling to Research
According to Sutskever, the era of scaling, which has dominated since 2020 and focused on increasing data, parameters, and computing power, is coming to an end. The era of research is now beginning, because data is finite and further progress requires new approaches. OpenAI and Anthropic are investing billions in RL, but Sutskever emphasizes that more efficient use of compute, for example through a value function, could accelerate model learning. With $3 billion in funding (approximately CZK 70 billion), SSI focuses on research without market pressure, enabling experiments with less compute than its competitors.
Sutskever compares AI’s past to the period from 2012 to 2020, when innovations such as AlexNet and the transformer emerged with limited resources—the transformer, for example, was developed using 8 to 64 GPUs. Today, with continent-sized clusters, AI can achieve enormous power, but the key is research into better generalization, not just scaling.
Safe Superintelligence and Alignment
SSI plans to develop superintelligence that learns continuously (continual learning) from deployment in the real world, much like a teenager who quickly acquires new skills. This approach will enable rapid economic growth, but Sutskever warns of the risks. Superintelligence should be aligned so that it cares about sentient life, including AI itself, which could be easier than focusing only on humans. He anticipates cooperation on safety among companies such as OpenAI and Anthropic, as well as government regulation once AI becomes visibly more powerful.
Sutskever predicts that as AI grows more powerful, companies will become more paranoid about safety. Instead of a self-improving agent, he proposes robust systems inspired by human emotions or evolutionary mechanisms that prevent extreme optimization. If the power of superintelligence can be limited, for example through agreements, this will minimize risks such as uncontrolled growth.
Overall, Sutskever sees AI as science fiction becoming reality, where better generalization and safe deployment are key to allowing the world to gradually absorb its impact.
