Imagine getting a new colleague. You do not have to explain anything to them because they have watched millions of hours of footage showing how people use computers and simply figured it out. That is exactly what a small team from San Francisco operating under the name Standard Intelligence set out to create. The result? A model called FDM-1 that has turned the world of computer agents upside down.
FDM-1 and why existing approaches are not enough
Until now, computer AI agents were developed like this: a language model with visual perception was taken, fine-tuned on screenshots manually annotated by specially trained people, and then taught specific tasks through reinforcement learning. That sounds reasonable, but there is one enormous catch.
The largest publicly available dataset for computer agents contains less than 20 hours of footage. Twenty hours. Meanwhile, tens of millions of hours of videos showing film editing, programming livestreams, game walkthroughs, and other content in which people use computers are available online. Existing models simply could not take advantage of this wealth of data. Moreover, they could handle only a few seconds of context, making more complex, long-term tasks unattainable for them.
Standard Intelligence took a different approach. It trained FDM-1 on 11 million hours of screen recordings. This is comparable to what GPT-3 did with text from the internet. But video is a different challenge than text.
How does the model know what the user was doing when it can only watch them? This is where the so-called inverse dynamics model (IDM) comes in. It works simply: if the letter "K" appears on the screen, someone most likely pressed the "K" key. The IDM was trained on 40,000 hours of manually annotated footage and then automatically labeled the remaining millions of hours of data. Without this step, the entire project would not have been possible.
A custom video encoder
The biggest technical challenge, however, lay elsewhere. Conventional models consume enormous numbers of tokens when processing video. Anthropic's Claude can handle approximately 162 frames in a 200,000-token window. FDM-1 can handle more than 48,000. That is roughly one hundred times more than OpenAI and over one hundred times more than Claude.
How did they do it? They created a custom video encoder trained with a masked compression objective, which learned to distinguish between areas of the screen containing a lot of information and those containing very little. The result is impressive: a 1-million-token context window can fit almost two hours of video at 30 frames per second. This opens the door to tasks that take tens of minutes, such as 3D modeling or complex financial analysis.
What FDM-1 can actually do
The team presented specific demonstrations of the model's capabilities, and each one deserves attention.
In Blender, FDM-1 independently creates a gear using precise mouse movements and keyboard shortcuts. No blind clicking, no mistakes caused by misunderstanding the interface. The model simply knows what it is doing.
After less than an hour of fine-tuning on real-world data, FDM-1 drives a car in San Francisco. Yes, you read that correctly. A model trained primarily on computer screens transferred its skills to the physical world and successfully navigated turns. That is exactly the kind of generalization AI researchers dream about.
The third demonstration may be the most practical: software testing. FDM-1 explores an application like an experienced tester and looks for bugs. In a demonstration involving a banking application, it found a security flaw that made it possible to submit a transfer twice and overdraw the account. This would normally take hours of manual testing.
The results are also supported by remarkable technical infrastructure. The team built a system that runs more than one million test runs per hour across 80,000 virtual machines. Each virtual machine runs a minimal Ubuntu desktop, and a single H100 GPU can control 42 of them simultaneously. The latency from screen capture to the model's action? Just 11 milliseconds. That is fast enough for the model to respond naturally, without any noticeable delay.
Standard Intelligence openly states that it believes artificial general intelligence will emerge within the next decade. FDM-1 is its contribution toward ensuring that this happens responsibly and with an understanding of how AI actually interacts with computers. Computer agents are ceasing to be toys and becoming genuine collaborators. And so far, FDM-1 is the most compelling evidence that this path is heading in the right direction.
