Artificial intelligence is beginning to perform real commissioned work at a level that a client would accept and pay for. This is shown by the latest results of the Remote Labor Index, published by the Center for AI Safety together with Scale Labs. The best of the models tested, Anthropic's Fable 5, was able to independently complete 16.1 percent of real-world projects. When the test was created last fall, the most capable AI managed only 2.5 percent. In less than eight months, the benchmark has thus increased fourfold. It is evidence of how quickly development is advancing.
The Remote Labor Index does not track answers to test questions. It focuses on finished work that someone commissioned and paid a professional to complete. The test includes 3D modeling, CAD, architecture, graphic design, video and animation, audio, data analysis, and web application development.
Each project includes a client brief, input files, and a so-called gold standard, meaning an output produced by a paid expert. The AI receives the same brief, and its work is then assessed by people. They compare it with the professional's work and decide whether it is equally good or better. The share of such projects forms the test's main metric, the automation rate.
The three latest models surpassed everything the test had seen so far. Fable 5 reached the aforementioned 16.1 percent, roughly double the 8.3 percent achieved by Opus 4.8. OpenAI's GPT-5.5 finished at 6.3 percent. For comparison, the best previously published result belonged to Opus 4.6, which reached 4.17 percent. When the test was launched, the leading models were still hovering around 2.5 percent. The jump is therefore particularly sharp for Fable.
There is one complication with Fable. The evaluators managed to review 218 of the total 240 projects before U.S. authorities restricted access to the model. The remaining twenty-two projects are distributed evenly across fields and difficulty levels. Even if Fable 5 failed every one of them, its automation rate would still be 14.6 percent, higher than that of any other model.
What the models created
Behind every number was a real commission. One of them asked for an engagement ring to be redesigned. The client wanted the center stone's emerald cut replaced with a marquise cut and requested an updated 3D model as well as photorealistic visualizations in rose and yellow gold. Fable 5's design looks significantly better than the outputs of older models. On closer inspection, however, it remains unprofessional, for example because of the overly simplistic rounding of the prongs.
Another assignment involved an approximately sixty-second flat 2D animation for a tree care company. The AI received only a voice-over and was asked to use it to create an advertisement introducing viewers to the company's process. The newer models showed a visible improvement in both image quality and the synchronization of the animation with the audio.
The third example fell within the field of architecture. A scanned cadastral plan, photographs, and dimensions were to be turned into clean, dimensioned floor-plan documentation, alternative furniture layouts, and photorealistic visualizations of a renovated bathroom. The floor plans are more accurate and the 3D models more detailed with the newer models, with Fable 5 once again the strongest among them. An interesting issue arose with GPT-5.5. Its attractive-looking visualization was actually a fake produced by an image generator. The actual 3D model it submitted was misshapen and lacked textures.
AI work still has to be assessed by people
As models become more capable and testing becomes more expensive, a tempting shortcut presents itself: replacing human evaluators with an automated judge, meaning another artificial intelligence. The test's authors built such a judge. It is an agent that opens both outputs in actual programs, reviews them as a client would, and decides whether it would accept the AI's work. They fine-tuned it on older models for which people had measured an aggregate rate of 3.3 percent.
But as soon as they deployed the same judge on the two latest models, which it had never seen, it significantly overshot the mark. For GPT-5.5, people measured 6.25 percent, while the automated judge reported 17.9 percent, nearly three times as much. For Opus 4.8, people arrived at 8.33 percent, while the judge gave it 18.8 percent.
The automated judge did rank the models correctly, placing both of the strongest models clearly at the top. But it grossly overestimated how much better they were. It is therefore useful for tracking relative progress, but not as a replacement for human assessment of actual capabilities.
The reason goes deeper. Evaluating such a project is itself a demanding task. It means opening files in the correct programs, knowing how to operate them, and assessing the result through the eyes of a client. Yet interacting with graphical interfaces is precisely what today's AI agents struggle with most. The judge therefore inherited the same weaknesses as the workers it grades. GPT-5.5's fake visualization illustrates this well. Detecting it requires opening the 3D project and examining the actual geometry, which a judge unable to operate the software properly simply will not do. OpenAI reached a similar conclusion. Its own automated evaluator agrees with human experts less often than the experts agree with each other.
Tools and capabilities available to the models
To ensure that the test measured the models' actual capabilities, the authors deployed them in environments commonly used by developers. Anthropic's models ran in Claude Code, while OpenAI's models ran in Codex CLI. Both environments were supplemented with a built-in computer-control tool. The agent takes a screenshot, clicks or types text, and then takes another screenshot. This allows it to operate graphical programs, not just the command line.
Each model was given access to a fully equipped Linux computer with more than thirty professional programs. These included Blender, FreeCAD, and OpenSCAD for 3D and CAD; GIMP, Inkscape, and Scribus for design; Kdenlive and ffmpeg for video; Audacity, LMMS, and MuseScore for audio; and the entire LibreOffice suite and LaTeX for documents.
The authors also took care to ensure that weak settings did not underestimate the models' capabilities. For each project, a model had up to 24 hours of real time, as well as one NVIDIA A100 graphics card when the task required it, for example for rendering or simulation, and the highest reasoning setting.
Using a pair of agents also proved useful. A worker tends to be too lenient toward its own work and rarely reviews it critically. Its work was therefore independently assessed by a critic, which opened files, took screenshots, and compared the result with the brief. The worker then revised the work until it satisfied the critic or exhausted its budget. The budget was $50 per project, or $150 for the more expensive Fable 5.
Longer work does not mean harder work for AI
In programming, a simple rule applies. The longer a task takes a person, the harder it is for AI to handle. The authors wanted to find out whether the same principle also applied across a diverse range of commissioned work. It does not.
The models' success rate does not decline based on how long the work took a professional. Many other factors determine the outcome. This corresponds to the idea of a jagged frontier of capabilities. Some tasks that an experienced person can complete quickly remain beyond reach, such as music transcription or real-time game testing. Conversely, today's models can dash off in minutes work that would take a person hours, such as digital art or code.
The automation rate grew from 2.5 to 16.1 percent in less than a year. Yet today's AI still fails to reach professional quality on most projects. Not one of Fable 5's three sample outputs would have passed as finished work. However, remote commissioned work encompasses a wide range of activities for which people are paid, and it is precisely here that the bar is rising rapidly.
Source: safe.ai/blog
