robots.txt in practice: the syntax, five example files and what it cannot do

Spencer Thursfield · Last reviewed 2026-09-12 · Evidence: RFC 9309, Google Search Central, operator documentation and our own robots.txt census

robots.txt is a plain text file at the root of a website that tells crawlers which addresses on the site they may fetch. It is a request that well-behaved crawlers honour rather than a lock, and it governs crawling, not whether a page appears in search results.

The rules are written down in RFC 9309, the Robots Exclusion Protocol, published by the IETF in September 2022 to formalise a method Martijn Koster first defined in 1994. Which AI crawler does what is covered separately in crawler controls: search versus training.

What is robots.txt?

It is one file per host, named robots.txt, at the top of the site: https://www.example.com/robots.txt. A copy in a subfolder is not read, and the file covers only the host and protocol it sits on, so it does not govern m.example.com or the http:// version of the site. Required

Inside, rules come in groups. Each group starts with one or more User-agent lines naming the crawlers it addresses, followed by Allow and Disallow lines holding paths. Google reads four fields (user-agent, allow, disallow and sitemap), treats anything after a # as a comment, and ignores other fields such as crawl-delay. Anything no rule disallows is allowed.

Write it in a plain text editor and save it as UTF-8: Google warns that word processors can add characters, such as curly quotes, that trip crawlers up.

What does a basic robots.txt look like?

For most small businesses, this is the whole file:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The asterisk addresses every crawler, and Allow: / lets them fetch the whole site. The Sitemap line must be a full address including https://. This site serves exactly this file.

You do not strictly need a robots.txt at all. When the file is missing, Google treats the 404 as permission to crawl everything.

How do you block a folder in robots.txt?

User-agent: *
Disallow: /drafts/
Disallow: /shop/
Allow: /shop/gift-cards/

Paths start with /, match from the start of the address and are case-sensitive, so /drafts/ does not cover /Drafts/. The trailing slash matters. Google's own examples show /fish matching /fish.html and /fishheads, while /fish/ matches only what is inside the folder.

When an Allow and a Disallow both match, the longer, more specific path wins. So the gift-card pages stay open while the rest of the shop is closed. If the two are equally specific, the standard says Allow should win. Required

How do you disallow all crawlers?

User-agent: *
Disallow: /

Two lines close the site to every crawler that obeys the protocol. Reddit serves exactly this. In our census of the 32 most-cited domains in our own measurement data, Reddit was the most-cited non-video domain, at 216 citations, with this file in place. We read that as access moving into commercial agreements, an association in one sample rather than proof of cause.

For a business without such an agreement, these two lines are the quickest way to disappear. On a staging copy, Google recommends password protection instead.

Should you block AI crawlers?

Decide by purpose. The operators publish separate names for training collection and for search, and each name can be addressed on its own:

# Keep content out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else, including AI search
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

OpenAI states that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot, and that each setting is independent. Anthropic states that restricting ClaudeBot signals that future material should be excluded from training, while Claude-SearchBot is the agent that indexes for search. Google-Extended covers training future Gemini models and grounding answers in Gemini Apps. Google says it does not affect inclusion or ranking in Google Search. Official recommendation

Blocking is rarer than the advice suggests. Twenty-eight of the 32 domains in our census permit all nine AI agents we tested. Medium is the clearest case of a by-purpose policy: it disallows GPTBot and ClaudeBot and permits the search and user-fetch agents. The folklore ledger rates both “block the AI crawlers to protect your content” and “open your robots.txt and you will get cited” as context-dependent. Context-dependent

Does robots.txt stop a page being indexed?

No. Google states that robots.txt is not a mechanism for keeping a web page out of Google. A disallowed page can still be indexed if other sites link to it. Google will not read the content, but the address, and anchor text from links pointing at it, can still show in results. Official recommendation

To keep a page out of results, use a noindex tag, password protection, or remove the page. The trap is combining the two: Google can only see a noindex on a page it is allowed to fetch. Disallow the page as well and the tag is never read.

Nor is robots.txt security. RFC 9309 states that its rules are not a form of access authorisation, and that listing paths in the file makes them publicly discoverable. A line reading Disallow: /private-client-files/ tells every visitor where to look. Required

What are the common robots.txt mistakes?

• Launching with the staging file. A Disallow: / written while the site was being built, still there after it went live.
• Naming a crawler and forgetting the rest. A crawler obeys only the most specific group that matches it and ignores the others. Add a User-agent: Googlebot group and Googlebot stops reading the rules under User-agent: *.
• Blocking a page you want removed. It hides the noindex that would have removed it.
• A file that errors. A missing file is harmless. One that returns a server error is not: for the first 12 hours, Google stops crawling the site and keeps retrying the file.

How do you check your robots.txt?

Open /robots.txt on your own domain in a browser. You should see plain text, not an error page or your homepage. Read every Disallow line and check you meant it.

Then open the robots.txt report in Google Search Console. It lists the files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors. It also lets you request a recrawl after an urgent fix. It works only for a Domain property or a URL-prefix property without a path.

An allow in robots.txt does not prove crawlers get through: a firewall or bot-protection setting can refuse them first. Can AI reach your site at all? walks through all five layers.