AI Crawler Access: The Complete robots.txt Guide for GEO

A blocked AI crawler is a site that cannot be cited by that crawler's engine, no matter how good the content is. If robots.txt disallows GPTBot, nothing you write will show up as a ChatGPT citation, because the page was never fetched in the first place. This is the highest-urgency fix in GEO because it isn't about quality — it's a wall you built yourself, and it's usually one line away from coming down.

What each AI crawler actually does

Not all AI crawlers do the same job, and conflating them is how well-intentioned robots.txt files end up blocking citations by accident.

GPTBot is OpenAI's crawler for gathering training data for future models. OpenAI also operates a separate agent for live retrieval used by ChatGPT's search and browsing features. If your goal is to appear as a citation in a ChatGPT answer, the retrieval-facing agent matters as much as GPTBot — blocking one and not the other only solves half the problem, and blocking both takes you out of both training and live answers.

ClaudeBot is Anthropic's crawler, used both for training data collection and, depending on the product surface, for fetching pages referenced during a conversation. Blocking it removes any chance of a page being read or cited by Claude.

PerplexityBot is Perplexity's crawler. Perplexity's product is built around live retrieval and citation — it fetches pages largely to answer specific queries and cite sources inline, so this is one of the more directly citation-relevant crawlers to keep open.

Google-Extended is a token that controls whether Google can use your content for Gemini model training and AI features like AI Overviews, separately from ordinary Google Search indexing (which Googlebot handles regardless). If Google-Extended is disallowed, your pages can still rank in classic search but are excluded from the data Google uses to generate AI answers.

CCBot belongs to Common Crawl, a nonprofit that publishes a huge, open web archive. CCBot itself doesn't power any single AI product, but its archive is a widely used training source for many language models built by organizations that don't run their own crawler. Blocking CCBot doesn't just affect one engine — it affects an unknown number of downstream models trained on the Common Crawl corpus.

Why this affects AI citations

Retrieval-augmented engines — ChatGPT with browsing, Perplexity, Gemini with grounding — work by fetching candidate pages, reading their content, and quoting or paraphrasing what they find. That fetch step is gated by robots.txt. If the relevant agent is disallowed, the engine's crawler respects the block and simply never reads the page. No content evaluation happens, no quality comparison against competitors happens — the page is out of the running before anything else about it is judged.

This is different from training-data blocks, which affect a model's baseline knowledge but not its ability to browse and cite a page live. A lot of robots.txt files were edited in 2023, during the first wave of anxiety about AI companies training on web content, and blocked every AI-sounding user agent as a blanket precaution. That reflex made sense as a training-data decision. It's a much worse decision now that citation-driving retrieval traffic runs through many of those same agent names. A rule written to opt out of model training can end up opting a site out of being recommended by that same company's product.

Is a Disallow: / under a named agent different from a Disallow: / under *?

Yes, and the difference matters here. A rule scoped to a specific user-agent, like:

User-agent: GPTBot
Disallow: /

...only blocks that one crawler. Other bots are unaffected. But a wildcard rule:

User-agent: *
Disallow: /

blocks every crawler that doesn't have its own more specific rule elsewhere in the file — including AI crawlers, unless they're given their own explicit Allow group. Robots.txt parsing generally prefers the most specific matching user-agent block, so a bot-specific Allow group can override a wildcard Disallow, but only if that specific group actually exists in the file. If your file only has a * block, every AI crawler is silently caught in it.

How to check

Fetch your live robots.txt directly — https://yourdomain.com/robots.txt — and read every User-agent block, not just the first one. Look specifically for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot by name, and separately check whether a wildcard * block disallows everything with no bot-specific override.

The CitationsAI Inspector extension flags this automatically: it checks the live robots.txt of any page you're auditing against the known AI crawler user agents and tells you which ones are blocked, so you don't have to manually cross-reference bot names against your file every time.

Beyond the file itself, check server or CDN access logs for the crawler's user-agent string. A permissive robots.txt with zero matching log entries over a reasonable window tells you the crawler either hasn't gotten around to your site yet or is being blocked somewhere else — a firewall rule, a bot-management product, or a CDN security setting that ignores robots.txt entirely and blocks by user-agent or IP range at the network layer.

How to fix it

The minimal fix is an explicit allow-list. Add (or correct) these groups in your robots.txt, keeping any existing rules for other bots intact:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you currently have a wildcard block for *, you don't need to remove it to fix AI crawler access — a specific User-agent block for a named bot takes precedence over the wildcard for that bot, per the robots.txt specification described at robotstxt.org. What you do need to verify is that each AI crawler you want to allow has its own explicit group, because relying on the wildcard's absence of a rule is fragile and easy to break with a future edit.

Training vs. retrieval: should you allow both?

Some organizations offer more granular control than a single user-agent toggle — for example, distinguishing an agent used for model training from one used for live retrieval and citation. If you're uncomfortable with your content training future models but you do want to be citable in live answers, check the specific platform's crawler documentation for whether it separates these two functions under different user-agent names. Where that separation exists, you can disallow the training-focused agent while allowing the retrieval-focused one. Where a platform uses a single agent for both purposes, the choice is binary: allow it and accept both uses, or block it and lose both.

Whatever you decide, write it down as a real, current decision rather than inheriting a block someone else added years ago for reasons that may no longer apply. Re-check this file whenever a new AI product launches a named crawler — the list of agents worth allowing is not fixed, and yesterday's complete allow-list can quietly go stale.

A note on verification cadence

Robots.txt is a static file, but crawler behavior isn't static — companies rename agents, split training from retrieval, and launch new bots. Treat a robots.txt audit as a recurring five-minute check, not a one-time fix. Re-fetch the live file after any redeploy that might have reset it to a framework default (some site generators regenerate robots.txt from a template and can silently drop custom allow rules), and glance at crawler log activity every month or so to confirm the bots you've allowed are actually showing up.