The three kinds of AI crawler, and only one means a human asked

If you look in your server logs right now, you will find requests from things called GPTBot, OAI-SearchBot and ChatGPT-User. They all have OpenAI behind them. They are not the same event, they do not mean the same thing, and if you count them together you will throw away the single most useful piece of information in the log.

There are three classes. The difference between them is who or what is waiting for the response.

Class 1: training crawlers

These collect text to train or refine a model. GPTBot, ClaudeBot, CCBot (Common Crawl, which feeds a great many training sets), Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai.

Nothing is waiting. A training fetch sends you no visitor and produces no citation. Its payoff, if it has one, arrives months later and is impossible to attribute: the model may end up knowing your product well enough to name it without looking anything up. That is worth something. It is just not measurable, and it is not traffic.

Two names in this class are worth pulling out, because they trip people up constantly. Google-Extended and Applebot-Extended are not crawlers at all. No request ever arrives with either as its user-agent. They are control tokens you put in robots.txt to opt out of Gemini and Apple Intelligence training, while the actual fetching keeps being done by Googlebot and Applebot. If you are looking for them in your logs you will wait forever.

Class 2: retrieval and index crawlers

These build the index an AI search product draws from when it answers. OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot, DuckAssistBot.

Still nobody waiting, but the meaning is completely different from class 1. This is the leading indicator of whether you are eligible to be cited. An engine that has never fetched your pricing page cannot cite your pricing page. If you are trying to appear in AI answers and you have no class 2 activity at all, that is the finding: you are not in the running yet, and no amount of writing more posts changes it until the retrieval crawler shows up.

This is also the class where blocking has a real, immediate cost. More on that in what to put in robots.txt for AI engines.

Class 3: user-triggered fetches

ChatGPT-User, Perplexity-User, Claude-User. These do not run on a schedule. They fire when a person asked a question, seconds ago, and the assistant went to your page to answer them.

A class 3 hit is not a crawl. It is a buyer, mid-question, with an assistant reading your page out loud to them.

Consider what that single log line tells you that your analytics cannot. Someone was far enough along to ask a real question. An assistant judged your page relevant enough to fetch in real time. And the answer that person received was shaped by whatever was on that page at that moment. You will never see the question and you will usually never see the visit, because the human reads the assistant's summary and may never click through.

"ChatGPT fetched your pricing page four times today to answer live questions" and "your site got 4,000 crawler hits this month" are not the same sentence. The first one tells you what your buyers are trying to find out. The second one tells you your robots.txt is permissive.

The full table

What to actually do with this

Count the three classes separately, or you have learned nothing. A single "AI traffic" figure moves mostly with class 1, which is the class that means the least, so the number that matters gets buried by the number that does not.

Treat class 2 as a readiness check and class 3 as a lead. No class 2 means you are not eligible for citations yet. Class 3 on a specific page means buyers are asking about that specific topic, and that page is currently your answer to them. Go and read it as if you were the person who asked.

Watch for the first class 3 hit on a page you just shipped. That is the moment a page stopped being content and started being infrastructure for someone else's decision.

The limits, stated honestly

This is log analysis, so it inherits every weakness of log analysis.

A user-agent is a claim, not a fact. Anyone can send User-Agent: GPTBot. Confirming a bot is genuine means reverse DNS or checking the operator's published IP ranges. For measuring your own site that mostly does not matter, because nobody has a reason to spoof traffic to you. It matters a great deal the moment somebody has an incentive to inflate a number.

robots.txt is a request. The major operators honour it. Bytespider has a long record of not doing so, and Meta's compliance is inconsistent. If you need a bot stopped rather than asked, that is a rule at your edge or WAF, not a line in a text file.

Google's AI Overviews are invisible here. They are assembled from the regular Google index by Googlebot, so there is no distinct user-agent to count and Search Console exposes no bot dimension at all. Anyone selling you AI Overview attribution from log data is guessing.

Class 3 volumes are small. These are single fetches tied to single human questions, not crawls. Four in a day on one page is a real signal. If you are used to reading traffic charts with thousands on the axis, you will dismiss the most valuable number on the page because it looks too small to matter.