More than half of all internet traffic today does not come from humans. Data from Ahrefs shows that 63% of websites receive visits from AI agents. Yet most website operators have no idea what is happening on their sites. Google Analytics may simply record a bot as a real customer.
AI agents are not just the web crawlers people remember from the days when they could easily be stopped with an entry in the robots.txt file. They are sophisticated tools that click buttons, fill out forms, and go through the entire purchasing process. Just like a human. And that is precisely why they are so difficult to catch.
AI Agent Activity on a Website
First, it is important to understand just how diverse this group is. The term "AI agent" covers several types of traffic that differ fundamentally from one another.
AI search engine crawlers such as Perplexity or Google AI Overviews browse your pages so they can include the content in AI search results. Then there are crawlers for training language models, such as OpenAI's GPTBot, Anthropic's ClaudeBot, or CCBot. They collect text to train or update models, and most of them identify themselves directly in the request header.
Another category is scrapers. These are third-party tools that steal content for competing products, monitor your prices, or copy text on a large scale.
Then there are agents acting on behalf of users. A customer tells ChatGPT, "compare prices from three online stores for me," and ChatGPT sends an agent that browses your pages, retrieves the information, and returns the result. Perplexity Comet, Amazon Buy for Me, and Base44 even complete purchases and reservations on behalf of users.
And then there are fraudulent agents that deliberately set out to cause harm. They test stolen payment card numbers, create dozens of fake accounts to exploit promotions, or map system vulnerabilities. An Accenture survey found that 78% of chief technology officers at financial institutions expect an increase in fraud caused by AI shopping agents.
Traditional Tools Are Not Enough to Stop Agents
Many teams believe they already have adequate protection. They enabled Bot Fight Mode in Cloudflare, set up CAPTCHA, and consider the problem solved. But the reality is different. The security team at cside deployed AI agents against two major bot detection platforms. They went undetected in 81 out of 100 cases, which is considered a serious failure of basic defenses.
Why is that? Traditional tools look for patterns that were valid a few years ago. They monitor IP address reputation, TLS fingerprints, or simple CAPTCHA challenges. But AI agents can handle AI challenges faster and more reliably than humans. And tools such as Playwright now offer a stealth mode that suppresses the navigator.webdriver flag, spoofs Canvas and WebGL fingerprints, and removes traces of the Chrome DevTools Protocol.
Meanwhile, Playwright is downloaded by more than 35 million developers per month. Google searches for the term "stealth browser" are reaching all-time highs. Locally run agents are even more difficult to handle. Someone runs automation through a browser extension on their MacBook. There is no data center IP address, just a normal residential IP, a normal browser fingerprint, and a normal-looking device.
How to Actually Detect AI Agents
There are essentially three approaches, each of which captures a different segment of traffic.
Server logs are the easiest place to start. A large proportion of AI crawlers identify themselves in the User-Agent string. GPTBot says "GPTBot," ClaudeBot says "ClaudeBot." If you use a CDN such as Cloudflare, Vercel, or Netlify, you can find the data directly in the dashboard. If you run your own server, you can download the logs from Nginx or Apache. The problem? This only works for agents that choose to be honest. The dangerous ones do not introduce themselves.
Analytics tools such as Google Analytics or PostHog are not sufficient for bot detection on their own. Crawlers do not execute JavaScript at all, so they never appear in analytics. Browser-based agents, on the other hand, generate data that looks like a human visit. Nevertheless, traces can still be found in analytics, such as an unusual increase in traffic from a specific Chrome browser, visits from countries you have geoblocked in Cloudflare (cside recorded traffic from China despite having blocked it in Cloudflare), or a sudden influx of thousands of sessions with the same screen resolution and from the same geographic area. These are signs of a bot farm.
Specialized AI agent detection tools are the third and most thorough option. They work similarly to analytics. You add a JavaScript snippet to your website, and the tool begins monitoring traffic across four layers simultaneously.
Four Signals to Watch
Specialized detection tools such as cside monitor four categories of signals at the same time. Only their combination provides a reliable result.
Identity is the first layer. Who does it claim to be? The User-Agent string, crawler signature, and cross-checking against databases of known bots. As mentioned, this only works for the honest ones. As much as 98% of AI traffic comes from major platforms such as OpenAI, Anthropic, Google, or Meta, and they usually identify themselves.
The network layer is the next consideration. Every request comes from an IP address associated with an autonomous system (ASN). Data center ASNs from AWS, GCP, or Azure are a strong sign of automation. TLS fingerprints reveal inconsistencies: if an agent identifies itself as Chrome but the TLS handshake looks like a Python script, the detection system will expose it. The same applies to geographic inconsistencies. An IP address from Frankfurt, but the browser's time zone is set to Shanghai and its language to zh-CN.
The browser and device layer looks for artifacts left by automation tools. Playwright and Puppeteer control the browser through the Chrome DevTools Protocol and leave traces—for example, the cdc_ prefix in window objects. WebGL, Canvas, and the Audio API should tell a consistent story about the device. If the browser reports a powerful GPU through WebGL but the Canvas output does not match, or the AudioContext fingerprint is entirely absent, it is a clear sign that someone has manipulated the browser environment.
Behavior is the final signal. Typing speed, time between navigations, form-filling patterns, mouse movement, and scroll depth. Even sophisticated automation produces patterns that can be detected. For example, one of cside's engineers analyzed three popular tools and found that one clicked every button exactly in the center, another always clicked slightly to the right of center, and the third clicked in the center but occasionally added a deliberately random off-center click. This was an attempt to introduce noise. Each of these patterns is detectable.
To Block or Not to Block?
This is where many people make a mistake. Automatically blocking everything that looks automated sounds logical. But customers now send agents to shop for them. Block a Perplexity Comet agent that was completing an order for a user, and you have lost a sale. The "bot or not" approach is outdated. A better question is: what did this agent come here to do?
If an agent tested 17 payment cards in three minutes, it is fraud. If an agent creates dozens of accounts at once, it is abusing a loyalty program. If an agent browses product pages and compares prices for a customer, it is probably legitimate traffic that you want to support.
Why the Web Is Not Going Anywhere
Occasionally, someone claims that websites will move to APIs and MCP anyway, and the browser will cease to be relevant. However, Google has published a guide on preparing websites for agents that explicitly addresses visual user interface optimization. If Google believed websites would operate purely through APIs, it would not offer advice on making buttons clearer and stabilizing page layouts.
Researchers from Carnegie Mellon University also found that hybrid agents combining web browsing with API calls significantly outperformed API-only approaches. In 77.7% of tasks, agents used both methods, and even when an API was available, they still returned to the browser.
Agents browse the web like humans because it simply works best. And that is precisely why they are visible at the browser level. It is also where they are easiest to catch.
