AI Coding Agents: Extensive Test Reveals Their Capabilities and Limitations

Timothy B. Lee decided to test the capabilities of today’s AI coding tools with a practical experiment. Instead of simple sample tasks, he chose a complex real-world project. As testing material, he used data on collisions involving Waymo vehicles—autonomous cars that statistically have fewer accidents per mile traveled than human-driven vehicles.

The assignment was clear: combine information from two different databases—one provided by Waymo and the other by the US National Highway Traffic Safety Administration. Then use the merged data to build a functional web application that would allow users to search, sort, and browse records of autonomous vehicle accidents.

During testing, Lee deliberately played the role of a complete programming beginner. He did not inspect the source code or attempt to diagnose technical problems. He relied exclusively on communicating with the AI agents through ordinary natural-language instructions.

Performance of Individual AI Coding Tools

Platforms That Struggled the Most

Bolt.new gave up right at the start with the error message "looks like you've hit the size limit". According to the documentation, the platform notifies users when the project size exceeds the available context window—200,000 tokens for free accounts or 500,000 for premium versions. The test files were too large to process. While competing solutions have features for progressively navigating large files, Bolt.new lacks such capabilities.

Replit completed the planning stage and then spent 17 minutes working on the implementation. Although it claimed the website was finished, the result could not be launched at all. Even repeated attempts to fix it failed—Lee was unable to get the platform to resolve the fundamental functionality issue.

Lovable created a visually impressive website but struggled to load the data. Through a series of instructions, Lee managed to get it to import most of the information and convert it into the required format. Nevertheless, there was still a problem with dates, which were displayed as "N/A" despite numerous requests to fix them.

Tools with Mixed Results

Windsurf started promisingly—it created a basic functional website faster than the others and allowed some modifications. However, the system proved unstable. A request for a drop-down menu to filter by injury severity (minor, moderate, serious, fatal) led to the creation of an interface that was unable to actually filter the data, even after several attempts to fix it.

OpenAI Codex completed the assignment, but its execution had shortcomings. While other tools intelligently selected key information such as the location, time, and severity of the incident, Codex included every available data column. The result was a page many times wider than a standard browser window. Even after time was invested in improving its appearance, various formatting issues and imperfections remained.

The Most Successful Solutions

Cursor delivered solid performance, although the beginning was not promising. Setting up the development environment took a long time and required several unsuccessful attempts. The first version of the site did not work at all—it displayed only an endless loading indicator. After some encouragement, however, Cursor resolved the issue and delivered an attractive-looking application. Most subsequent modification requests were fulfilled on the first attempt.

Claude Code achieved the best results. It handled the task almost seamlessly, with minimal complications. After creating the basic structure, it was able to implement the requested features and design changes quickly and accurately, often on the first attempt.

Key Observations from Practical Testing

Lee was surprised by the amount of repetition and number of attempts required when using these tools. It was common for him to report an error, for the agent to attempt a fix, and for the problem to persist. Sometimes this cycle was repeated for five to ten consecutive attempts, with the AI trying different approaches.

At the same time, it became clear that some errors remained unsolvable regardless of the number of attempts. Lee observed this behavior with the Lovable, Replit, and Windsurf platforms.

A clear trade-off emerged between ease of use and functionality. Bolt, Lovable, and Replit present themselves as tools for "intuitive coding" that allow non-programmers to create complete websites with a single command. However, all three fail when faced with more complex or unconventional requirements.

By contrast, Claude Code and Codex target professional programmers. They require API keys and knowledge of Unix commands, which may discourage beginners. However, they compensate for this with significantly greater versatility and performance.

Using AI Coding Tools in Practice

Lee spoke with Aaron Votre, a developer at Bright Harbor, a company focused on disaster recovery. Votre makes extensive use of Cursor and Claude Code and reports dramatic productivity gains thanks to these tools.

"We maintain an extensive file containing hundreds of lines of our guidelines," Votre explained. This document, permanently incorporated into Claude Code’s context, specifies the software tools used for various purposes and contains advice that the company would give to a new employee.

The guidelines include instructions such as "resolve every issue before continuing to ensure code quality", "sort aliases alphabetically", and "test formatting functions separately for better isolation".

"Whenever it makes a mistake, we add it," Votre noted. "Claude is currently the best choice for us because we have the best setup for it."

Another strategy, according to Votre, is to have the agent develop a detailed plan before implementation. A human programmer can review and modify the plan—this prevents problems caused by unclear or confusing instructions and provides additional context.

English as the New Programming Language

Lee refers to a statement made by Andrej Karpathy several years ago: "The hottest new programming language is English."

The history of programming shows a gradual shift toward higher levels of abstraction. While the first programmers had to work with low-level mathematical operations, the 1950s brought languages such as Cobol and Fortran. Modern languages such as Python include extensive libraries that make it possible to create powerful programs in just a few lines.

AI coding agents represent the next stage of this evolution. Instead of writing instructions in C++ or Python, programmers can enter requirements in English, and the agent converts them into the appropriate programming language.

This paradigm eliminates the need to deal with implementation details and search for minor errors. What has not changed, however, is that someone must define what the computer should do and provide sufficiently precise instructions. For large projects, this requires systematic thinking, an understanding of trade-offs, attention to detail, and a deep understanding of how computers work.

Lee anticipates similar developments in other professions affected by AI agents. There will be legal assistants helping with documents and contracts, but someone must assign tasks and evaluate the results—which requires professional expertise. Rather than replacing professionals, these will be new tools for their work.