Blog · Sep 4, 2026
752 sites refused our scanner: what a 403 or 401 means for AI visibility
5.6% of the sites we tried to scan refused the connection. That may mean AI crawlers are being turned away too.
Key takeaways
- 752 of 13,342 attempted sites (5.6%) refused our scanner: 449 with a 403 Forbidden error and 303 with a 401 login request.
- A refusal to our scanner is a signal, not proof, that AI crawlers are refused as well.
- The usual causes are firewall and bot-protection settings, region or IP rules, and accidental site-wide login walls.
What 403 and 401 mean
- 403 Forbidden: the server recognized the request and refused it, usually because of a firewall, security plugin or bot-blocking rule.
- 401 Unauthorized: the server asked for a login before showing anything. Some sites require login for the whole site by accident, often from a membership or "coming soon" plugin left running.
Both are easy to miss: the site loads normally for the owner, who is logged in or on an allowed network, while automated visitors are turned away.
What it implies for AI crawlers
We can say what happened to our scanner. We can't say for certain what happens to each AI crawler, because firewalls treat different crawlers differently, and some verify crawlers by their IP addresses. But a site that refuses automated visitors in general is a good candidate for refusing AI crawlers too, and it is worth checking.
Why it happens
- Aggressive bot protection. Services and plugins such as Cloudflare's bot settings, Wordfence and Sucuri can classify crawlers as threats. "200 attacks blocked this week" may include legitimate visitors.
- Region or IP rules. Rules meant to stop spam can catch crawler IP ranges, which are spread around the world.
- Accidental login walls. Membership, client-portal or coming-soon plugins left active across the whole site.
- Tight rate limits. A crawler fetching several pages quickly can trip limits designed for human browsing.
- Misconfigured hotlink or CDN protection that blocks every non-browser user-agent.
How to check
- Logged-out test. Open your site in a private window. If it loads normally, a login wall is unlikely, but bot-specific blocking is still possible because firewalls often allow browsers and block crawlers.
- Scan test. Run a CrawlReady scan. If it can't get in, something is refusing automated visitors.
- Ask your host or developer whether the server, CDN or a security plugin blocks automated requests, and whether the AI crawlers you want are allowed.
The fix
- Firewall and bot protection: add the AI crawlers you want to allow to your allowlist or exceptions. Their user-agent names are published by each vendor, so check their current documentation. On Cloudflare, review the bot settings under Security.
- Login walls: confirm any "require login," maintenance or coming-soon plugin is off on the live site, or that it exempts public pages.
Send to your web person: > Our site may be refusing automated visitors: a scanner received 403 or 401 errors. Please check our firewall, CDN, security plugin and bot settings, and make sure the AI crawlers we want (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) are allowed.
The bottom line
Every other fix, whether FAQ, metadata or structure, assumes crawlers can reach your pages. A hard block is the most absolute problem we found, and it can't be seen from the front of the site.
About this data: 13,342 sites attempted in October 2026; 449 returned 403 and 303 returned 401, counted among the 2,973 that could not be scored.