Runway has introduced Praxis-1, its first robotics world action model, built on large-scale video pretraining. The company proposes fine-tuning that foundation on robot-specific data to reduce the need for demonstrations collected through teleoperation. Early partner tests are underway, with the model’s weights planned for public release in the coming months.
From visual motion to robot commands
The proposed training process has two stages. Runway first aims to use web video to teach motion, interactions between objects and physically plausible behavior, followed by fine-tuning on robot data. Its goal is to reduce the demonstrations needed for individual tasks and hardware configurations.
Video footage alone usually lacks the information a robot records while performing a task. Web videos generally contain no control commands, joint positions or force measurements. Praxis-1 therefore requires robot-specific fine-tuning to connect learned visual concepts with commands the machine can execute.
An experiment comparing two pretraining sources
Runway reports that model performance improves as it scales up pretraining on third-person video. In one placement experiment, the company compared web video with footage of teleoperated robots. After fine-tuning, the web-video-pretrained version recorded a final placement error of 16.1 cm, compared with 16.0 cm for the version pretrained on robot video.
Another result concerns the evaluation of control policies in simulation. Runway reports a 0.95 correlation between evaluations in its generated simulations and results on physical robots.
Targeting difficult manipulation tasks
Runway is targeting four manipulation challenges with Praxis-1. One is selecting the correct target among nearly identical objects. Another involves cluttered scenes where objects overlap and obscure one another.
For transparent objects, the company is targeting shape and depth estimation when visual cues are weak. The fourth area involves deformable objects, such as cloth, that lack fixed grasp points.
Tests across robot platforms and planned open weights
Runway’s announcement describes the same control policy operating across different robot configurations without retraining, including bimanual systems, six-degree-of-freedom arms and mobile bases. In early partner tests, Noble Machines is using a bimanual manipulation system, Standard Bots is using its RO1 six-degree-of-freedom arm, and Ultra is using a mobile robot base.
The company demonstrated transfer between environments in a separate example. The same control policy operated in both a controlled studio and a domestic kitchen without additional training.
Runway plans to release Praxis-1’s weights publicly in the coming months. Developers can meanwhile request early access from the company’s robotics team.
