Developers today add skills to AI agents almost automatically, assuming that providing a described procedure will fill in the agent's missing knowledge and enable it to perform better. A new study by several universities, which conducted more than 8,000 tests, shows that this assumption is wrong. According to the study, skills do not serve as a repository of knowledge, but rather as a guardrail that prevents the agent from going off track in the middle of a complex task.
How researchers tested skills
Developers usually build skill libraries through trial and error based on their own intuition, but the study's authors wanted to isolate the effect of skills, so they had agents work on tests where tasks required several consecutive steps. They used Terminal-Bench 2.0 and SkillsBench, which require running commands, debugging errors, and then verifying whether the result actually matches the assignment.
They then compared three ways of working, all based on the same prior experience. In the first mode, the agent received the task without any context from previous attempts; in the second, it was given raw, unorganized logs from a previous attempt; and in the third, it was provided with a distilled skill in the form of a cleaned-up and consolidated description of the procedure extracted from those same logs.
The difference between the second and third modes was 6.06 percentage points in favor of the distilled skill, confirming that experience alone is not enough and that the form in which the developer packages it before passing it to the agent matters. Anyone can repeat this test and verify whether their own abstractions offer anything beyond simply inserting raw logs into the prompt.
Workflows win, explanations lose
The study finds that skills work when unorganized logs are transformed into procedural guidance that stabilizes the agent's execution of the task, meaning that a skill should show the agent the individual steps rather than teach it the facts themselves.
The numbers confirm this: procedural guidance accounted for 65.7 percent of the cases in which a skill helped, while the transfer of specific knowledge contributed to success in only 4.5 percent of cases. Skills therefore do not fix the model's high-level reasoning, but make it more robust during execution itself. This is most apparent in environment failures, where the agent struggles with tool configuration, works around dependencies, or is unable to prepare the environment. The share of such failures fell from 5.3 percent when running without guidance to just 0.2 percent among agents with a distilled skill.
The study describes one specific case in which an agent without guidance knew how to write a fix in React but failed the response-time check. The interface loaded in 915 milliseconds even though the limit was under 800 milliseconds, because the agent ran requests sequentially rather than concurrently. The skill then served as a workflow that told the agent to rewrite independent operations using Promise.all, start them as early as possible, and wait for the result only at the end. When the agent followed this sequence, it passed all tests in 11.74 seconds.
The clear advice for developers is not to write skills as explanations of subject matter or general algorithms, but instead to design them as standardized step-by-step checklists that the agent must follow.
The danger of unlabeled experience
However, the authors warn that skills fail when they are based on assumptions, do not fit the context, or cannot adapt, with many of these problems arising during their creation. When a skill is distilled from raw logs and no one tells the agent which processes succeeded and which failed, the resulting procedure quickly falls apart.
The researchers demonstrated this in an experiment where they deliberately removed this information and presented a Gemini-based agent with a mixed batch of five logs containing three successful and two unsuccessful attempts. When the agent could see how the individual attempts had turned out during distillation, it achieved a 74.6 percent success rate on subsequent tasks. But once this label was removed, the success rate fell to 40 percent. Without information about the outcome, it could not separate useful signal from noise and permanently recorded its own hallucinations and inefficient steps in the skill library as well.
Logs have value, but only when they contain the right signals, and their quality can be pre-filtered, for example, by having a language model evaluate them as an independent judge.
The strange mathematics of catalog search
The second problem emerges as the library grows, because when the number of available skills increased from five to one hundred, the precision of the retrieved skills that were actually used fell from 29.6 percent to just 3.3 percent. You might expect task success to collapse along with this decline, but that did not happen: for the Gemini-based agent, the success rate remained stable at roughly 36 to 39 percent even when it had one hundred options to choose from.
The explanation is that retrieving exactly the one correct skill is neither a necessary nor a sufficient condition for success. Although agents often reach for the wrong skills, even a related skill can provide enough useful guidance for them to complete the task. If, instead of retrieving the exact procedure for debugging a checkout interface, the agent retrieves a general procedure for debugging interfaces, it still receives a meaningful checklist that it can successfully follow.
The real risk of large catalogs lies specifically in semantic interchangeability, because once the library accumulates skills that are virtually identical in vector space, search can no longer distinguish between them and begins returning random results.
How to stop the agent from retrieving whatever it feels like
The solution proposed by the authors does not rely on smarter vector search, but on an architecture in which skills are first divided by domain, after which the conditions for triggering them are defined. In practice, this means building a two-tier system where, at the first tier, a router based on a language model classifies the task into a specific category, for example according to whether it involves frontend or backend work. At the second tier, the agent retrieves a skill only when a strictly defined condition is met, such as an exact error string in the terminal log, for example a connection refused message.
The researchers add that working with skills must be understood as a matter of the entire lifecycle, not as a one-time insertion of memory into the prompt. You should therefore treat your skills as a library of operating procedures that you regularly review and verify that the agent can use correctly, and build an architecture around them that circumvents semantic interchangeability before it begins to cause harm.
