Blog · May 16, 2026
Is my site blocking ChatGPT?
Blocking fails silently. Your site looks normal to visitors while AI crawlers are turned away at one of two doors.
Key takeaways
- Blocking can happen in robots.txt and, separately, at your server or firewall. Checking only one can miss the problem.
- OpenAI uses different crawlers for training (GPTBot), ChatGPT search (OAI-SearchBot) and user requests (ChatGPT-User). Check all three.
- Most blocks are accidental: a leftover staging rule, a security plugin, or a firewall default.
The short answer
Possibly, and you wouldn't know. When a crawler is blocked, nothing breaks visibly. Your site looks fine to visitors, but assistants may stop reading your pages and your business can drop out of their answers.
Two doors, not one
- The sign on the door: robots.txt. The file at
yoursite.com/robots.txttells crawlers which parts of your site they may visit. ADisallow: /under a crawler's name means "stay out." - The bouncer: your server. Even when robots.txt says "come in," your host, firewall or a bot-protection tool can turn the crawler away, often without any message to you. Some tools treat AI crawlers as scrapers by default.
Many owners check robots.txt, see nothing wrong and assume they're fine, while the server may be blocking some or all AI crawlers.
Check the sign yourself
Open yoursite.com/robots.txt and search for these names:
- GPTBot: OpenAI's training crawler.
- OAI-SearchBot: the crawler behind ChatGPT search.
- ChatGPT-User: fetches pages when a person asks ChatGPT about a link.
A Disallow: / next to any of them blocks that crawler. Blocking GPTBot alone doesn't stop ChatGPT search, and OpenAI's documentation notes that ChatGPT-User is handled separately from its robots.txt tags. Check OpenAI's current crawler documentation for details. Also check for a wildcard rule: User-agent: * followed by Disallow: / blocks every crawler.
Check the bouncer
Ask your host or developer whether any firewall, CDN or security plugin has bot protection switched on, and whether AI crawlers are allowed. You can also test from a terminal with a crawler's user-agent:
`` curl -I -A "OAI-SearchBot" https://yoursite.com/ ``
A 200 response is a good sign. A 403 or 401 means the server refused the request. This test only imitates the user-agent name. A firewall that verifies crawlers by their IP addresses may treat the real crawler differently, so treat it as a signal, not proof.
How blocks happen by accident
- A
Disallow: /copied from a development template and never removed at launch. - A security or SEO plugin with "block AI bots" enabled by default.
- A firewall rule that groups AI crawlers with malicious scrapers.
The reliable way to know
You need both checks: what robots.txt says and what the server does, for each major AI crawler. CrawlReady runs both for every scan and shows the results side by side, so a mismatch is obvious at a glance.