Why Progress in Artificial Intelligence Is Not About Innovation, but Data
Artificial intelligence has made incredible progress over the past fifteen years. This progress may seem inevitable—even though truly groundbreaking discoveries are rare, AI continues to advance steadily. Some researchers have even proclaimed a "Moore's Law for AI," according to which computers' ability to perform certain tasks increases exponentially over time. Every year, our AI systems become smarter, faster, and cheaper, with no end in sight.
Research Successes Versus Stagnation
The research community at MIT, Stanford, CMU, and industry giants such as Meta and Google has made significant advances. For example, Stanford introduced FlashAttention in 2022, Google developed speculative decoding in 2023, the Muon optimizer emerged in 2024, and DeepSeek released DeepSeek-R1 in 2025.
Yet the latest models, such as Grok 3 and GPT-4.5, have improved only marginally.
Four Key Breakthroughs in AI
If we look at genuine paradigm shifts, we find four key moments:
1. Deep neural networks: AlexNet won an image recognition competition in 2012.
2. Transformers + LLMs: Google proposed transformers in the paper "Attention Is All You Need" (2017), which led to BERT (Google, 2018) and GPT (OpenAI, 2018).
3. RLHF: First introduced in OpenAI's InstructGPT paper in 2022.
4. Reasoning: OpenAI released O1 in 2024, followed by DeepSeek R1.
Old Ideas, New Data Sources
It is not unreasonable to claim that all the fundamental mechanisms behind these breakthroughs already existed in the 1990s. Supervised learning grew out of Claude Shannon's work in the 1940s, while reinforcement learning can be traced back to policy-gradient methods from 1992.
The key point, however, is that each breakthrough unlocked a new source of data:
1. AlexNet unlocked ImageNet—a large database of labeled images.
2. Transformers unlocked training on the entire internet.
3. RLHF made it possible to learn from human evaluations of "good text."
4. Reasoning uses "verifiers" such as calculators and compilers.
Data vs. New Ideas
Researchers have found that different architectures (transformers vs. state-space models) achieve similar results when trained on the same data. This suggests that there is an upper limit to what we can learn from a given dataset. No amount of training tricks can circumvent the fact that there is only so much that can be learned from a given dataset.
The Future: Video and Robots
Our next paradigm shift will not come from new algorithms, but from new data sources.
Video: About 500 hours of footage are uploaded to YouTube every minute—far more data than all the text on the internet. Videos contain not only words, but also intonation and information about physics and culture. Google will likely begin training models on YouTube as soon as they become efficient enough.
Robots: Systems for collecting data from sensors and cameras could be the next major source, provided we can process the massive influx of data from robots.
It is difficult to predict whether YouTube, robots, or something else will be the Next Big Thing. But if we want to make progress in AI, perhaps we should stop looking for new ideas and start looking for new data.
Author Jack Morris points out in his article that progress in artificial intelligence is driven not by new algorithms, but rather by new datasets. An analysis of four key moments in AI history shows that each breakthrough was enabled by access to a new type of data, not by revolutionary new technology.
