Blog · Apr 23, 2026
robots.txt and AI crawlers: what to allow and what to block
Not every AI crawler does the same job. Here is how to tell them apart and set your robots.txt on purpose.
Key takeaways
- AI crawlers fall into three groups: training, search, and user-initiated. Blocking one group doesn't block the others.
- A single
User-agent: *rule withDisallow: /blocks every crawler at once, and it is often left over from a staging site. - Decide deliberately: for most businesses, allowing the search and user-initiated crawlers is what keeps you eligible to be cited.
The problem
Many sites block AI crawlers without knowing it. A staging rule such as Disallow: / gets copied to the live site, or a security plugin blocks "bots" too aggressively. Either way, the assistants can't read your site and can't mention it.
Three kinds of AI crawler
| Type | What it does | Examples |
|---|---|---|
| Search | Builds the index behind an assistant's search and citations | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User-initiated | Fetches a page when a person asks about it | ChatGPT-User, Claude-User, Perplexity-User |
| Training | Collects content to train models | GPTBot, ClaudeBot, CCBot |
| Opt-out controls | Tokens that govern AI use of content by large platforms | Google-Extended, Applebot-Extended |
Vendors document these separately, and the details change, so check each vendor's current crawler documentation before you decide. Two cautions from published documentation: OpenAI's documentation separates ChatGPT-User from its robots.txt tags, and Perplexity says Perplexity-User generally ignores robots.txt because a person triggers it. Blocking those two in robots.txt may not stop them.
OpenAI runs three of them
OpenAI alone fields three crawlers, and blocking one does not block the others:
| Crawler | What it does | What blocking it means |
|---|---|---|
| GPTBot | Collects public content to train OpenAI's models | Opts your content out of future model training |
| OAI-SearchBot | Builds the index behind ChatGPT search | Can keep you out of ChatGPT search results |
| ChatGPT-User | Fetches a page when a person asks ChatGPT about it | Can stop ChatGPT reading a link someone gives it |
Whether to block any of them is a trade-off. For most local businesses, allowing OAI-SearchBot and ChatGPT-User keeps you eligible to be read and cited; whether to allow GPTBot is a separate decision about training. For publishers protecting paywalled or licensed content, blocking training crawlers is a reasonable step. Most blocks, though, are not decisions at all: a broad "block all unknown bots" rule, an outdated blocklist, or a leftover Disallow: / catches AI crawlers along the way, and the owner never learns a specific crawler was affected.
Audit your robots.txt in 30 seconds
Open yoursite.com/robots.txt in a browser. Look for two things:
User-agentlines naming any crawler above, followed byDisallow: /.- A wildcard block:
User-agent: *followed byDisallow: /. This blocks everything, including AI crawlers and search engines.
The business decision
Blocking is a legitimate choice, but it is a tradeoff:
- Allow the search and user-initiated crawlers if you want assistants to be able to cite and recommend you. For most businesses, visibility wins.
- Opt out of training separately if you don't want your content used to train models. This is a different decision from being cited.
- Block everything only if your content is the product, such as paywalled journalism or proprietary data.
Example rules
Allow the crawlers that support citations:
``` User-agent: OAI-SearchBot Allow: /
User-agent: ChatGPT-User Allow: /
User-agent: Claude-SearchBot Allow: /
User-agent: Claude-User Allow: /
User-agent: PerplexityBot Allow: / ```
Opt out of training while staying visible in answers:
``` User-agent: GPTBot Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: Google-Extended Disallow: / ```
Google-Extended doesn't affect how Google Search indexes your pages.
The wildcard trap
`` User-agent: * Disallow: / ``
This is the rule we see most often behind accidental blocking: meant for a staging site, copied to production, and forgotten. If your site is live and this is in your robots.txt, fix it today.
How CrawlReady checks it
Every scan fetches your robots.txt and tests each major AI crawler individually, not just the wildcard. The report shows which crawlers are blocked and the rule responsible, and also tests whether your server lets each one in.