Artificial intelligence, as most people know it today, writes, translates, programs, and answers questions. It does this so well that it is easy to forget its limitations. A large language model can perfectly describe what happens when a glass falls off a table, but it cannot simulate the fall itself.
That is precisely why part of the field is shifting toward a new concept. These are so-called world models, or large action models, whose goal is to do for robotics what ChatGPT has done for writing and programming, as Christopher Mims described in the WSJ at the end of August. These models are not trained on literature and code, but on video games and simulations. In some situations, they can already navigate three-dimensional space and independently manipulate objects according to simple instructions.
How a language model works
A large language model (LLM) is a neural network trained to predict the next token in a sequence. After processing a sufficient amount of text, it can generate coherent and contextually appropriate language. At the core of its mechanics is the statistical matching of patterns across vast quantities of text.
Models such as GPT, Claude, and Gemini are based on the transformer architecture. The attention mechanism evaluates how strongly each word in a sequence relates to all the others. The result is a system capable of writing, summarizing, translating, and reasoning about language tasks. However, it retains nothing from each individual response, so it has no memory unless you explicitly provide it with the conversation history, and it maintains no state of its environment.
Where LLMs fall short
Research repeatedly returns to the main weakness: the lack of grounding in reality. A language model has no internal understanding of physical cause and effect. It can describe what will happen because it has seen similar texts millions of times, but it does not calculate the laws of physics in any way.
One example from research illustrates this best. A language model can discuss chess fluently and accurately, but after a while it will suggest moving a piece that is not even on the board. It did not learn to track the position, only what commentary on a chess game looks like. Analysts at Forrester add that these models’ ability to solve logic problems changes dramatically even with a slight adjustment to word order. This reveals that their reasoning is based on patterns rather than on the underlying structure of the problem.
Kent Rollins of General Intuition, who previously led the development of Fortnite at Epic Games, approaches the issue from another angle. According to him, text describes the world only with significant information loss because the world existed long before writing, and words necessarily omit some of what matters. Animals have been developing brains to model their surroundings for about half a billion years, while language is, by comparison, a novelty only around one hundred thousand years old.
A world model imagines the future
The world model concept represents a learned internal understanding of how an environment works. The model receives a description of the current state and a planned action, then predicts the future state. The goal is not to generate sentences, but to accurately estimate the consequences.
Robotics experts have known this for a long time. The control systems of more advanced machines rely on physical simulations of their environments, but these were programmed manually and are highly specialized. What works for one type of robot may fail for another. New companies instead want to develop control systems that are as universal in operating machines as today’s language models are in writing text.
The most frequently cited evidence that this approach works is the DreamerV3 system. A study published in the journal Nature in April 2025 showed that a single algorithm can handle more than 150 different tasks. In Minecraft, for example, it can learn from scratch how to mine diamonds, entirely without human demonstrations or task-specific tuning.
The key is that the agent imagines trajectories within the learned model. It simulates thousands of possible futures, evaluates them, and adjusts its behavior based on the results. Most of the learning takes place inside the model rather than in the real world. This is the source of a tenfold to hundredfold improvement in sample efficiency compared with conventional reinforcement learning, where the agent must learn from actual actions in the environment. This represents an enormous advantage in fields where every real-world attempt is expensive or dangerous.
Two distinct architectures
While language models are built on transformers and the parallel processing of sequences, world models use different building blocks. Older systems such as PlaNet and Dreamer combined recurrent networks with latent dynamics models. Yann LeCun’s JEPA architecture learns abstract representations instead of predicting individual pixels, which significantly reduces computing requirements. Meta trained the V-JEPA model on one million hours of internet video and subsequently adapted it for robotic planning with a relatively small amount of additional data.
The fundamental difference can be summarized in a single sentence. A transformer is stateless on each pass, while a world model retains its state and updates it with every action. For a language model to process an image or sound, it must convert them into tokens just like words. With a three-dimensional scene, however, this approach has proved so inefficient that the resulting models are generally too slow to control a robot.
Data is the most expensive part
Text is available online almost free of charge and in unlimited quantities. Pairs of actions and consequences, however, are much more difficult to obtain because recordings from real robots are expensive. Nvidia therefore offers the Cosmos platform, trained on 20 million hours of real-world footage from transportation, industrial operations, and robotics. Companies such as Wayve, 1X, and Figure AI are building on it, while Uber and Waabi use it to simulate autonomous driving.
General Intuition turned to a different source. It uses millions of hours of gameplay from real people through its sister service Medal.tv and records not only every frame of the game but also every button press. The model thus connects an action with its consequence instead of merely watching video. After brief fine-tuning, it can control a robot in the real world much as it would control a character in a game. At its offices in New York and Geneva, the company demonstrates this with a four-legged robot that only needs to know where it is, what kind of body it inhabits, and what it is supposed to do. The machine must be controllable with a gamepad or keyboard and transmit a live video feed, which rules out most bipedal humanoids.
Weaknesses
Small inaccuracies in state predictions accumulate over a long trajectory until the internal representation diverges completely from reality. Moving from simulation to the physical world requires careful fine-tuning, and a model trained on one type of environment often fails in another.
George Konidaris of Brown University points to a more fundamental difficulty. Although language models are growing larger and more capable, they still make mistakes, but hallucinations cannot be tolerated in a robot. According to him, the real world is full of exceptions and hard boundaries, and nothing in it can be missed only approximately because any collision involving the machine completely changes the entire calculation. Investor Moritz Baier-Lentz estimates that compared with language models, today’s world models are roughly at the level of GPT-2, a model released in 2019.
When to use each approach
If the task involves language, such as searching, summarizing, working with documents, or reviewing code, it belongs in the hands of a language model. However, if a system must act reliably in a physical environment where an error in dynamics costs money or endangers people, a world model comes into play.
Nevertheless, the most likely scenario is a combination of both approaches. The language model understands the assignment, plans the steps, and communicates with the user, while the world model tracks the state and verifies the plan before an irreversible step. A more modest option is also possible, in which the language model calls a world model as a tool, much as it uses a calculator or search engine today. Meanwhile, Google DeepMind has introduced the Genie 3 system, which generates interactive 3D environments in real time at twenty-four frames per second.
Sources: wsj.com, kanerika.com and fenxi.fr
