AI Teaches Robots to Move Using Video Game Footage

AI Teaches Robots to Move Using Video Game Footage

Ondřej Barták
Ondřej Barták
Entrepreneur and Programmer
9. 7. 2026
5 minutes reading
AI Teaches Robots to Move Using Video Game Footage

    Startup General Intuition announced that it has raised $320 million. The money will mainly go toward one goal: teaching artificial intelligence to navigate the real world by having it play video games. The company trains its models on hundreds of millions of hours of gameplay footage and believes that gaming data is a shortcut to enabling machines to acquire something approaching human intuition.

    A TechCrunch journalist described what it looks like right on the development floor of the company's New York office. Something resembling Fortnite was running on a monitor. But no human was controlling the game. "Our agent plays nonstop for a hundred hours," boasted Chief Product Officer Kent Rollins. A large four-legged robot then walked by. According to company co-founder and CEO Pim de Witte, the same brain powers both the robot and the gaming agent.

    Just eight minutes of street data was enough

    The robot behaved like a toddler who still does not fully understand where its body ends and its surroundings begin. It bumped into chairs, collided with a trash can, walked around a visitor, and continued down the hallway. It navigated using just a single camera, its only eye.

    Data analyst Josh Duplantis revealed a detail that shows where the company is headed. Fine-tuning the model for the four-legged robot required just eight minutes of real-world data. More interestingly, the data was not collected in the office where the robot was currently roaming, but outside on the street. The model was therefore able to transfer experience from one environment to a completely different one.

    It is not about the image, but about what the player pressed

    The company was spun out of another of de Witte's companies, called Medal. It allows players to record and share clips from their games. The hundreds of millions of hours of recorded gameplay served as the first batch of data on which the model learned what is known as spatiotemporal orientation, meaning how to move through space and time.

    But the key ingredient was not the footage itself. It was the action labels embedded in each clip, precise records of which button the player pressed and at what moment. De Witte says that most competitors merely try to infer actions from video, which he believes is not enough. "We see this as the next phase of model training," he explained. According to him, the same model responds both to events on the screen in Fortnite and to events in the real world, something no language model could do.

    World model

    De Witte then let the journalist try a so-called world model, a simulated environment that is not generated by a conventional game engine but created frame by frame. When testing similar simulations, it is common for the character you control to walk straight through a wall. Not here. From all those hours of gameplay, the model somehow learned that a wall is a wall, a ladder is for climbing, and shadows grow longer as the sun moves.

    What is interesting is how the company itself views this simulation. It is not a product it wants to sell. Internally, they call it the "gym" because it serves as a training ground. The company wants to sell the agent model itself. De Witte argues that the action labels embedded in gameplay help the model distinguish the "self" from the "environment," enabling it to better understand cause and effect.

    No matter how impressive the demo may look, General Intuition is not the only company trying to solve this problem. And one issue remains unresolved: no one has yet fully succeeded in deploying such a model in the physical world at scale. Most similar approaches require enormous amounts of real-world data, which is slow and expensive to collect. The company is therefore betting that playing video games offers a cheaper and faster path.

    No weapons or violence

    De Witte spent three years working in humanitarian aid, including with Doctors Without Borders. This experience also led him to draw a clear line regarding where the company's technology must not end up. No agents deployed to harm people. "We don't want to be the part of the system that escalates tensions," he said. At the same time, he added that he has no objection to his models being used in search-and-rescue operations. This stance comes at a time when Silicon Valley is increasingly embracing military contracts. De Witte is Dutch, and much of his team comes from Europe, which clearly shapes the company.

    The co-founder is also thinking about the people whom artificial intelligence will eventually put out of work. When he was young, he earned $1.5 million by building and operating a private server for RuneScape. The company recently launched Nerve, a kind of labor marketplace where players can earn money using their own equipment. They start with data labeling and can gradually move on to remotely operating robots. De Witte pointed out that Medal's users are precisely the generation at risk of being the first displaced from the labor market by artificial intelligence. He wants them to have a stake in what comes next.

    They want to provide the foundation for others

    For now, the company has a handful of customers in gaming, simulation, and robotics. De Witte describes its ambitions by comparing it to Anthropic or OpenAI. He wants to be a model provider on whose technology others can build. "We're not going to build a self-driving car company," he said. "We'll make it ten times easier for the next person to build a self-driving car company."

    The four-legged robot is currently the first physical body the company has tested in the real world. But it has also experimented with drones and other devices, including testing the model in racing games. "It works on anything you control with a game controller or a keyboard and mouse," de Witte noted. The goal is to set a data flywheel in motion by gradually bringing in customers who will provide the company with interesting real-world data, which will then advance its research.

    Category:Robotics
    Did you enjoy this article?
    Discover more interesting posts on our blog
    Back to blog

    Related posts

    China Unveils First Robotic Centaur. It Will Work Where Humans DieChina Unveils First Robotic Centaur. It Will Work Where Humans Die
    At a Shanghai exhibition center this July, a machine appeared that at first glance defied classification. Shanghai-based Run Robotics unveiled a robot at the WAIC 2026 conference that it calls
    3 min read
    27. 7. 2026
    Not for homes, but robots for industry. English firm Humanoid reaches billion-dollar valuationNot for homes, but robots for industry. English firm Humanoid reaches billion-dollar valuation
    London-based robotics company Humanoid, officially known as SKL Robotics, has joined the ranks of so-called unicorns—young companies valued at more than one billion dollars. According to...
    6 min read
    20. 7. 2026
    A Breakthrough in Robotics: Robots Get Artificial Hands That Work Like Ours!A Breakthrough in Robotics: Robots Get Artificial Hands That Work Like Ours!
    California-based 1X from Palo Alto has unveiled new hands for its NEO robot. They have 25 degrees of freedom, are tendon-driven like human hands and, according to the manufacturer, approach human dexterity, strength and reliability.
    5 min read
    14. 7. 2026
    Přihlaste se k odběru našeho newsletteru
    Zůstaňte informováni o nejnovějších příspěvcích, exkluzivních nabídkách, a aktualizacích.
    CodedTrip

    Operated by CodedTrip LLC, USA.

    YouTube
    TikTok