What to put in robots.txt for AI engines

Here is the mistake, in two lines:

User-agent: GPTBot
Disallow: /

A founder writes that because they do not want their writing used to train a model without compensation. Reasonable position. They then assume they have opted out of ChatGPT.

They have not. GPTBot collects training data. OAI-SearchBot builds the index ChatGPT search answers from. Two different crawlers, two different jobs, one letter of visual difference at a glance. The rule above blocks the one that was never going to send you anything, and leaves untouched the one that decides whether you appear in answers. Whether that is the outcome you wanted depends entirely on which one you thought you were blocking.

The inverse mistake is more expensive and much more common:

User-agent: *
Disallow: /

Usually inherited from a staging config that shipped, or from a "block the AI scrapers" snippet copied off a forum. It blocks retrieval too. Every AI search engine that would have cited you is now told not to look, and nothing in your analytics will ever tell you that is why you are absent.

Decide per class, not per company

There are three classes of AI crawler and they want different things. I go through them in detail here, but the short version is all you need to write a robots file:

So the decision is not "do I allow AI bots". It is two independent decisions: do I want to be findable in AI answers (allow retrieval and user-triggered, always, if you sell anything) and do I mind my text being used as training data (a content-rights preference, with no measurable traffic consequence either way).

What we chose, and why

This site allows everything except Bytespider.

# Retrieval: builds the index AI search answers from.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: DuckAssistBot
Allow: /

# User-triggered: a person is waiting. Never blocked.
User-agent: ChatGPT-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

# Training: allowed deliberately. Being in the weights
# means a model can name us without fetching anything.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

# ByteDance. Crawls aggressively, has a long record of
# ignoring this file, sends nothing back.
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.getmiravi.com/sitemap.xml

That is the live file, with the comments shortened. You can read the real one and check that this post is telling you the truth, which is a habit worth having about anybody's advice on this subject.

The training decision is the one worth explaining, because the common advice is to block. We allow it on purpose. When someone asks an assistant "what should I use to get my startup cited in AI answers", the best outcome is a model that answers from what it already knows, with no retrieval step at all. You cannot measure that, you cannot attribute it, and you get exactly one shot at being in a training set for any given crawl. For a company nobody has heard of yet, that trade looks good. For a publisher whose archive is the product, it looks terrible. Both are correct answers to different situations.

Bytespider is excluded not as a rights position but as a straightforward cost decision: it crawls hard, and there is no ByteDance surface where our buyers are asking questions.

Five things that will bite you

1. Disallow and noindex are not alternatives, and combining them backfires

Disallow says do not fetch this. noindex says fetch it but do not list it. So if you Disallow a page and put noindex on it, the crawler never fetches the page, never sees the noindex, and the URL can still surface from inbound links. If you want a page gone, let it be crawled and let it say noindex.

2. robots.txt is host-specific, and redirects will get you

The file must be at the root of the exact scheme and host being crawled. https://www.example.com/robots.txt and https://example.com/robots.txt are two different files as far as a crawler is concerned. If your apex redirects to www, make sure the redirect actually resolves for /robots.txt and that the file lands where the crawler follows to. This is a five minute check with curl and it is worth doing today:

curl -sI -A "OAI-SearchBot/1.0" https://example.com/robots.txt
curl -sI -A "OAI-SearchBot/1.0" https://www.example.com/robots.txt

3. It is a request, not a fence

The major operators honour robots.txt. Bytespider has a long record of ignoring it and Meta's compliance is inconsistent, which means the one rule in our file that we actually want enforced is the one least likely to be obeyed. If you need a crawler stopped, that is a rule at your CDN or WAF. A text file cannot do it.

4. Blocking at the CDN is invisible in robots.txt

Plenty of "AI bot blocking" toggles now ship enabled by default in CDN dashboards. Your robots.txt can say Allow: / to every retrieval crawler while your edge returns 403 to all of them. Check what actually happens on the wire, not what your config says should happen.

5. Allow-listing with User-agent: * Disallow: / underneath is fragile

Most crawlers match the most specific group that names them and ignore the rest, so a named Allow above a wildcard Disallow usually works. Usually. Parsers differ at the margins, and a bot you have not named yet gets the wildcard. If your intent is "everyone except X", write it that way: a permissive wildcard and explicit Disallow groups for the exceptions.

What about llms.txt?

It is a proposed convention: a Markdown file at your root describing your site and pointing at your important pages, so an assistant can orient without parsing your navigation. We publish one.

Be clear-eyed about it. No major engine has committed to reading llms.txt, and there is no evidence it currently affects whether you get cited. It costs ten minutes and it is a genuinely useful summary of your own site, which is reason enough. It is not a ranking mechanism, and anyone selling it as one is ahead of the facts.

The five minute version

Allow every retrieval and user-triggered crawler, without exception, if you sell anything. Make a deliberate content-rights call on training crawlers and know that it costs you no measurable traffic either way. Add your sitemap line. Then curl your own robots.txt on both hosts, as a bot, and confirm you are serving what you think you are serving. That last step is the one that finds the real problem.