Why Progress in Artificial Intelligence Is About Data, Not Innovation

Why Progress in Artificial Intelligence Is About Data, Not Innovation

Ondřej Barták
Ondřej Barták
Entrepreneur and Programmer
7. 7. 2025
3 minutes reading · 12 views
Why Progress in Artificial Intelligence Is About Data, Not Innovation

Why Progress in Artificial Intelligence Is Not About Innovation, but Data

Artificial intelligence has made incredible progress over the past fifteen years. This progress may seem inevitable—even though truly groundbreaking discoveries are rare, AI continues to advance steadily. Some researchers have even proclaimed a "Moore's Law for AI," according to which computers' ability to perform certain tasks increases exponentially over time. Every year, our AI systems become smarter, faster, and cheaper, with no end in sight.

Research Successes Versus Stagnation

The research community at MIT, Stanford, CMU, and industry giants such as Meta and Google has made significant advances. For example, Stanford introduced FlashAttention in 2022, Google developed speculative decoding in 2023, the Muon optimizer emerged in 2024, and DeepSeek released DeepSeek-R1 in 2025.

Yet the latest models, such as Grok 3 and GPT-4.5, have improved only marginally.

Four Key Breakthroughs in AI

If we look at genuine paradigm shifts, we find four key moments:

1. Deep neural networks: AlexNet won an image recognition competition in 2012.

2. Transformers + LLMs: Google proposed transformers in the paper "Attention Is All You Need" (2017), which led to BERT (Google, 2018) and GPT (OpenAI, 2018).

3. RLHF: First introduced in OpenAI's InstructGPT paper in 2022.

4. Reasoning: OpenAI released O1 in 2024, followed by DeepSeek R1.

Old Ideas, New Data Sources

It is not unreasonable to claim that all the fundamental mechanisms behind these breakthroughs already existed in the 1990s. Supervised learning grew out of Claude Shannon's work in the 1940s, while reinforcement learning can be traced back to policy-gradient methods from 1992.

The key point, however, is that each breakthrough unlocked a new source of data:

1. AlexNet unlocked ImageNet—a large database of labeled images.

2. Transformers unlocked training on the entire internet.

3. RLHF made it possible to learn from human evaluations of "good text."

4. Reasoning uses "verifiers" such as calculators and compilers.

Data vs. New Ideas

Researchers have found that different architectures (transformers vs. state-space models) achieve similar results when trained on the same data. This suggests that there is an upper limit to what we can learn from a given dataset. No amount of training tricks can circumvent the fact that there is only so much that can be learned from a given dataset.

The Future: Video and Robots

Our next paradigm shift will not come from new algorithms, but from new data sources.

Video: About 500 hours of footage are uploaded to YouTube every minute—far more data than all the text on the internet. Videos contain not only words, but also intonation and information about physics and culture. Google will likely begin training models on YouTube as soon as they become efficient enough.

Robots: Systems for collecting data from sensors and cameras could be the next major source, provided we can process the massive influx of data from robots.

It is difficult to predict whether YouTube, robots, or something else will be the Next Big Thing. But if we want to make progress in AI, perhaps we should stop looking for new ideas and start looking for new data.


Author Jack Morris points out in his article that progress in artificial intelligence is driven not by new algorithms, but rather by new datasets. An analysis of four key moments in AI history shows that each breakthrough was enabled by access to a new type of data, not by revolutionary new technology.

Advertisement

Content created with help from UpTier.

SEO and GEO on autopilot. UpTier’s multi-agent systems write and optimize content for search engines and AI answers.

Discover UpTier ↗

Category:AI
Did you enjoy this article?
Discover more interesting posts on our blog
Back to blog

Related posts

Google makes Tasks free for Gemini 4 Hydrogen usersGoogle makes Tasks free for Gemini 4 Hydrogen users
Tasks is becoming available for free to hundreds of millions of Gemini 4 Hydrogen users. Google says it can call businesses, book appointments and complete purchases on their behalf.
1 min read
18. 11. 2026
OpenAI introduces GPT-6.2 Astra and Sol for professional and everyday workOpenAI introduces GPT-6.2 Astra and Sol for professional and everyday work
OpenAI has introduced two models: Astra for complex professional tasks and Sol for everyday work. Both are available in ChatGPT Work, Codex and through the API.
3 min read
17. 11. 2026
Tencent releases Hy4 Flash with image support and a context window of up to 1 million tokensTencent releases Hy4 Flash with image support and a context window of up to 1 million tokens
The lighter Hy4 Flash model targets chat, documents and routine code edits. It accepts text and images, supports tool use and, according to Tencent, matched Claude Haiku 5.5 in tests of everyday tasks.
1 min read
16. 11. 2026
Přihlaste se k odběru našeho newsletteru
Zůstaňte informováni o nejnovějších příspěvcích, exkluzivních nabídkách, a aktualizacích.
CodedTrip

Operated by CodedTrip LLC, USA.

YouTube
TikTok