AI Agents: What They Can Really Do and Where Their Limits Lie

There is enormous excitement surrounding artificial intelligence and large language models (LLMs) in particular. However, discussions about how leading AI companies differ from previous generations of companies often take place at such an abstract level that they border on meaninglessness. It is similar to claiming that your company could be much better if it simply adopted more software—technically true, but not a particularly useful statement.

This approach attempts to briefly summarize how AI agents work, apply this summary to several real-world use cases, and generally argue that agents multiply the quality of your software and system design. If your software or systems are poorly designed, agents will only cause harm.

How AI Agents Work: Four Key Capabilities

1. Evaluating the Context Window Using an LLM

At its core, using an LLM involves an API call containing a prompt. For example, you might call Anthropic's /v1/message with the prompt: "How should I adopt LLMs in my company?" This prompt is used to fill the LLM's context window, conditioning the model to generate certain types of responses.

Prompt engineering, also known as context engineering, involves deciding what to put into the context window to best generate the responses you are looking for. For example, In-Context Learning (ICL) is a form of context engineering in which you provide a series of similar examples before asking a question.

2. Designing Tools and Enriching Context

An LLM does not actually call a tool by itself. There is a five-step tool-calling process:

1. The program designer must define a set of tools that the LLM can suggest
2. Each LLM API call includes these tools as options
3. The API response is either generated text or a recommendation to call a specific tool with specific parameters
4. The program decides whether and how to fulfill the requested tool use
5. If the program decides to call the tool, it invokes it and calls the LLM API with the tool's output

3. Controlling the Flow of Tool Use

Agents control the flow of tool use through rules or statistical analysis:

Flow control through rules:

• It may allow a given tool to be used only once in a particular workflow
• It may require human approval for parameters above a certain value (for example, refunds over 100 dollars)
• It may run a generated Python program and return the output for dataset analysis
• It may apply a permissions system to tool use

Flow control through statistics:

• If a refund amount is higher than 99% of other refunds, it may escalate it to a human
• If a user has used a tool more than 99% of other users, it may deny its use for the rest of the day

4. Agents as Software Programs

Agents can do everything software can do to create better context windows. This includes:

• Creating general context to add to the context window
• Initiating a workflow based on an incoming ticket
• Periodically initiating workflows at a specific time

Practical Example 1: Customer Support Agent

A typical customer support process has several tiers of agents who handle increasingly complex customer issues. The goal is to take over the simplest tier first and then gradually move on to higher tiers.

The approach could be:

• Allow tickets (or support chats) to flow to the AI agent
• Provide the agent with various tools for retrieving user information, escalating to the next support tier, refunding a purchase, and closing a user account
• Include customer support instructions in the context window
• Flow-control rules ensuring that all calls are escalated to a human if they are not resolved within a certain period

Importantly, even if you transition "customer support to AI agents," you still have:

• A tier of human agents handling the most complex calls
• People reviewing periodic performance statistics
• People performing quality assurance on AI agent-customer interactions

Practical Example 2: Triaging Incoming Bug Reports

When an incident is reported at your company or when you receive a bug report, the first challenge is to determine how serious the issue may be.

The process could work as follows:

• Route all newly created incidents and tickets to this agent for review
• Give the agent access to tools for opening an incident, retrieving current incidents, retrieving recently created tickets, and retrieving production metrics
• Redundant LLM providers for critical workflows—if the LLM provider's API is unavailable, it retries three times over ten seconds, then falls back to a second model provider (for example, Anthropic first, and if it is unavailable, OpenAI)
• Merge duplicates and assess impact
• Suggest the cause and apply known safe feature flags

Limitations and Important Observations

It is important to understand that agents cannot solve every problem. They cannot speed up a database restore beyond what the network bandwidth supports. Access to text-based assessment does not create missing tools. Nor does text-based assessment resolve access permissions or force nonexistent documents into existence.

Agents multiply the quality of your system design—well-implemented agents can make you significantly more efficient. Poorly implemented ones will only amplify your problems.

LLMs and agents are powerful mechanisms that will truly change how products are designed and how they operate. An entire generation of software creators and company executives is currently learning how these tools work.

Software is not magic—software is highly logical. What software can accomplish, however, is magical if we use it effectively. The combination of agents, excellent system design, and excellent software design is what will allow agents to truly shine.