Who is actually open to AI crawlers?
Correction 2026-08-07, hours after first publication
The first version of this page reported that Yelp permits the AI training crawlers while blocking the search ones. That was the opposite of the truth, and the fault was ours, not the source's. A hand-written script read the rules sitting between one User-Agent: line and the next; Yelp lists several agents consecutively above a single shared rule block, so the script saw an empty block and read it as permission. The evaluator we built for this study merges consecutive agent lines as the standard requires, and disagreed within hours of publication.
We also reported that six domains would not serve an honestly-identified research client. On re-measurement all six responded normally. That claim is withdrawn, not amended.
Everything below is the re-measured version, taken with the evaluator. The original wording is preserved in this page's history and in the ledger's revision log.
Standard advice says open your robots.txt to the AI crawlers so you can be cited. We read the robots.txt of the 32 most-cited domains in our own measurement data and evaluated each against the nine major AI agents. Almost none of them is closed — and the most-cited source of all is one of the few that is.
What we did
Across two complete cycles of our own visibility measurement we recorded just over two thousand citations spanning 476 distinct domains. We took the 32 most-cited, fetched each site's robots.txt with an honestly-identified client, and evaluated it per agent for a representative page path — nine agents covering the three purposes: training collection, search indexing, and live fetches made when a person asks a question.
The sample is the head of one client's cited set: questions a UK business's customers would ask before buying. It describes the sources these engines chose for these questions. It is not a sample of the web.
Three findings
1. Blocking the major AI crawlers is rare
Twenty-eight of the 32 permit all nine agents. Only four block anything at all. Whatever is deciding which sources get cited, it is mostly not robots.txt, because almost nothing in the cited set is closed. That is the finding that should temper the standard advice: you cannot gain an advantage by opening a door that your competitors have not closed either.
2. The most-cited source blocks everyone — and is cited anyway
Reddit is the most-cited non-video domain in our data, at 216 citations. Its robots.txt is four lines:
User-agent: *
Disallow: /No named AI agents. No allowlist for the AI companies it has publicly signed agreements with — because they do not need one. Access to the largest sources has moved off the public protocol and into commercial agreements. A robots.txt tells you nothing about a contract, and a contract does not read your robots.txt.
The lesson for a business without a contract is not that robots.txt stopped mattering. It is that robots.txt was never the lever — it is the floor. A page a crawler cannot fetch cannot be chosen. Being fetchable removes an obstacle; it does not create a reason to choose you.
3. Where publishers do differentiate, they differentiate by purpose
Two of the four blockers are doing something more considered than a blanket refusal.
Medium disallows GPTBot and ClaudeBot — the training collectors — while permitting OAI-SearchBot, PerplexityBot, Claude-SearchBot and the live user-fetch agents. Index me for answers; do not train on me. That is a coherent position, written down.
Yelp takes the opposite one: all nine blocked, with narrow Allow carve-outs for its editorial /article paths. The business listings — the user-generated content that is the actual asset — are closed to every AI agent.
Two publishers, two defensible and completely different decisions, neither of which survives being summarised as “allows AI” or “blocks AI”. The three purposes have different consequences and have to be counted separately, which is why we never sum them. See crawler controls: search versus training.
The part we cannot measure, and will not
Two of the 32 domains — Reddit and Yelp — forbid our instrument by robots directive. We record those as blocked, name them, and exclude them from analysis. We do not disguise our client as an ordinary browser to take the page anyway. It costs us data, and the refusal is the point: a study willing to break a stated rule to improve a number cannot then be trusted about the number.
A separate effect is real but unmeasured by us so far: some sites refuse a plain client and serve a browser. We will not put a figure on that until we have measured it properly, having just published one we had to withdraw.
What this does not show
- It is not causal. That the most-cited source is closed is an association in our data. The commercial-agreement explanation fits it; so might others.
- It is one sample, once. The head of a single client's cited set, read on 7 August 2026. Robots files change; ours are stored as they were on the day.
- Nine agents, one path. Rules are frequently path-specific — Yelp's carve-outs prove it — so a verdict for one page is not a verdict for a whole site.
- Our first attempt at this was wrong. See the correction above. The instrument caught the error, which is an argument for instruments over quick scripts, not for trusting either without a check.
Where this lands in the ledger
This research supports three entries in the folklore ledger: opening robots.txt as a route to citation (context-dependent), the ceiling on citation measurement (unsupported), and the measured section attached to blocking the AI crawlers.