Reading your crawler logs: the evidence you already own

Lindsay Smith · Published 2026-08 · Evidence: strongly supported; verification method documented by operators

Before any question about citations, a prior question has a definite answer sitting in your server logs: do AI systems fetch your pages at all? Ordinary analytics cannot see it, because crawlers skip the JavaScript that analytics depends on. The log can.

What to look for

Each fetch records a user agent. The ones that matter fall into the three purposes covered in crawler controls: search indexers (OAI-SearchBot and kin), training collectors (GPTBot and kin), and live user-requested fetches, where an assistant reads your page because a customer asked about you at that moment. That last category comes closer to a lead than anything else in crawl data.

Verify identity; user agents lie

Anyone can send any user-agent string, and scrapers impersonate the famous ones every day. The operators publish the IP ranges their crawlers use; a claimed OAI-SearchBot fetch from an address outside OpenAI's published ranges is an impostor. Keep verified, claimed-but-unverified, and unverifiable traffic in separate columns. Mixing them corrupts every conclusion downstream. Strongly supported

What crawl data means, and what it doesn't

Being fetched is a precondition rather than visibility. A thousand GPTBot visits predict nothing about citation; zero visits guarantee absence. Treat the log as the pass/fail gate at the bottom of the pipeline. Pages never fetched can never be cited, and a page you care about with no crawler history has a diagnosable cause.