Can AI reach your site at all?

Lindsay Smith · Published 2026-08 · Evidence: required, checkable

One question comes before any strategy: can the machines fetch your pages? When the answer is no, nothing tells you. The request fails at one of five layers and you appear in no answers. The failure reaches further than robots.txt, so work down the whole stack. Each check takes minutes.

The five layers

1. DNS and hosting. Does your domain resolve, everywhere, over both IPv4 and IPv6? A crawler on a schedule reads intermittent hosting as permanent absence.
2. The edge: CDN and firewall. This is the modern failure. Bot-protection products block AI crawlers by default, and a firewall that returns 403 overrules a robots.txt that says allow. Test by fetching as the crawler's user agent, from outside your network.
3. robots.txt. Allow the search crawlers you want by name; see search vs training access. One syntax error can block everything below the line.
4. Status codes. Real pages must return 200; moved pages one clean 301; dead pages an honest 404. Redirect chains leak retrievability, and so do soft 404s: a "page not found" message served with status 200. Required
5. Rendering. Is the content in the HTML the server sends, or assembled afterwards by JavaScript? AI search crawlers are, in general, worse at rendering than Googlebot. If your text exists only after scripts run, assume some systems never see it. See static HTML and JavaScript rendering.

Then verify in the logs

Your server logs hold the ground truth: which crawlers fetched which pages, and when. Verify claimed identities against the operators' published IP ranges, because anyone can fake a user-agent string. A page no AI crawler has ever fetched is a page no AI can ever cite. The log is where that silence becomes visible, and a page you care about with no crawler history has a diagnosable cause. Strongly supported