Blog · Aug 27, 2026
152 sites scored zero on crawler access: are you blocking by accident?
About 1 in 68 small business sites blocks three or more AI crawlers outright, and many owners probably don't know.
Key takeaways
- 152 of 10,369 scored sites (1.5%) block three or more AI crawlers and score zero on crawler access.
- The usual causes are a WordPress setting left on, a security plugin default, a copied template or a "temporary" developer block.
- You can check in 60 seconds, and fixing it is usually a one-line change.
The file you've probably never opened
Every site can have a plain-text file at yoursite.com/robots.txt that tells crawlers where they may go. AI companies' crawlers check it before visiting. Many robots.txt files were written years ago, copied from templates or generated by security plugins, and they can contain a rule like Disallow: / under a broad user-agent, or blocks for crawlers the owner has never heard of. One over-broad rule can lock out every AI crawler at once, while search engines keep visiting normally. The site looks fine, and only the AI crawlers are gone.
How it happens without anyone noticing
- "Discourage search engines" left on. WordPress has a checkbox for this under Settings > Reading, intended for sites under construction. It typically produces a block-everything robots.txt, and it often stays on after launch.
- Security plugin defaults. Some plugins block "unknown bots" aggressively, and newer AI crawlers can count as unknown. The owner sees "200 malicious bots blocked" and feels protected.
- Copied templates. A robots.txt copied from a forum post years ago may block crawlers by name, including ones that now matter.
- A "temporary" developer block. A crawler block added during a redesign is forgotten after launch.
Check yours in 60 seconds
- Open
yoursite.com/robots.txt. A 404 means no file, which means everything is allowed by default. - Look for AI crawler names (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, OAI-SearchBot) next to
Disallow. Blocking some of these is a legitimate choice, such as opting out of AI training, but make sure it was yours. - Look for
User-agent: *followed byDisallow: /. That blocks every crawler not explicitly allowed. - On WordPress, confirm "Discourage search engines from indexing this site" is unchecked.
The fix
Do it yourself (WordPress): uncheck the setting above, then open your SEO plugin's robots.txt editor (Yoast and Rank Math both have one) and remove any rules that block AI crawlers you want to allow.
Send to your web person: > Please review our robots.txt and confirm we aren't blocking AI crawlers by mistake. Remove any Disallow rules targeting them unless we've chosen to opt out. Also confirm WordPress's "discourage search engines" setting is off.
For which crawlers to allow and which to opt out of, see robots.txt and AI crawlers.
Why it matters
A zero on crawler access means AI assistants may not be able to read your site at all, however good the rest of it is. That's why it's the first thing our scanner checks.
About this data: 152 of 10,369 sites scored in October 2026 had a crawler-access score of zero, meaning three or more of the AI crawlers we test were blocked in robots.txt.