Crawler controls: search access is not training access
The most consequential robots.txt mistake of the AI era is treating "AI crawlers" as one thing. Operators run different crawlers for different purposes, and blocking the wrong one removes you from AI search without a word of warning.
The three purposes
- Search discovery. These crawlers feed AI search answers. OpenAI's is OAI-SearchBot: block it and ChatGPT search cannot surface you. Official recommendation
- Model training. These crawlers gather training data. OpenAI's is GPTBot. Blocking it is a legitimate policy choice, and it leaves your ChatGPT search visibility untouched. Official recommendation
- User-requested fetches. An assistant fetches a page because a user asked about it in the moment. Blocking these frustrates your own prospective customers.
The decision, plainly
For a small business that wants customers: allow search and user-fetch crawlers; decide training access on principle, never by accident. The widespread copy-pasted "block all AI bots" robots.txt files conflate the three purposes and opt businesses out of being recommended.
Verify, don't assume
A crawler line in robots.txt tells you what you asked for. It says nothing about what happened. Two checks settle it: confirm your CDN or firewall permits the crawlers your robots.txt allows (a common silent failure), and read your server logs for crawler visits, verified against the operators' published IP ranges, because anyone can fake a user-agent string. See reading your crawler logs.
What bernard does about this
Hosted sites get a managed robots.txt with search and user-fetch crawlers allowed by name, and identity-verified crawler logging as standard. You can see which operators fetched which pages. verified Training-access policy is your call; bernard implements either choice.