Most small businesses treat their robots.txt file as ground truth: "I allowed GPTBot, so I'm visible to ChatGPT. I blocked it, so I'm not." New research says that assumption is wrong for a meaningful share of sites — and the gap between what your robots.txt says and what your CDN or firewall actually does can quietly cost you AI citations either way.
HasData's AI Crawler Block Index, published in July 2026, measured actual crawler behavior — not just declared policy — across 10,894 domains (9,746 top sites from the Tranco list plus 1,148 news publishers). The core finding: 39.5% of GPTBot "Disallow" rules in robots.txt are never actually enforced by the site's own infrastructure. The site declares a block, but GPTBot crawls it anyway, because whatever CDN or web application firewall sits in front of the site was never configured to match the robots.txt policy.
The Reverse Problem Happens Too
The same gap runs the other direction. Site owners who explicitly allow AI crawlers in robots.txt — hoping to be citable in ChatGPT, Perplexity, or Google's AI features — can still get silently rejected at the network edge. A CDN's bot-management rule, a WAF challenge page, or a blanket "block AI bots" toggle in a hosting dashboard all run before robots.txt is ever read. If any of them classifies an AI crawler as suspicious traffic, the crawler gets a 403 or a CAPTCHA page — and your robots.txt permission never mattered. HasData's index found this enforcement is inconsistent by provider: on the sites it sampled, Fastly tends to hard-block flagged bots, Cloudflare serves a challenge page, and CloudFront generally waves traffic through — three different real-world outcomes from the same declared policy, depending only on which infrastructure sits in front of the site.
Blocking Doesn't Even Guarantee What You Think
The index also tested the flip side of the assumption — that blocking AI crawlers protects you from being cited. Across 10 test queries in Google's AI Mode, HasData found 52 cited domains that were also in its 10,894-domain sample; 27 of those 52 (51.9%) blocked at least one AI crawler in their robots.txt — roughly three times the block rate of the overall sample. In other words, declaring a block in robots.txt does not reliably keep a page out of Google's AI Mode citations, because AI Mode draws on Google's own index and crawl infrastructure, which operates independently of how a site treats OpenAI's, Anthropic's, or Perplexity's bots.
This lines up with what we flagged in last month's look at Cloudflare's September 15 crawler policy change: by HasData's estimate, that single deadline affects roughly 8.5% of the entire web, and the risk there was Googlebot itself — not just AI-specific bots — getting swept into a blanket "block Training crawlers" rule. The broader lesson from this new data is that Cloudflare isn't the only place this happens; it's a structural blind spot at the CDN/WAF layer across providers.
How to Actually Check — Not Guess
- Read your access logs, not just robots.txt. Search your server or CDN logs for the real user-agent strings — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot — and check the HTTP status code each one actually received. A string of 403s or CAPTCHA redirects tells you the truth your robots.txt file can't.
- Send a real request as the bot. From an external server (not your own network), send a request with an AI crawler's user-agent string and read the actual response code and body. If you get a challenge page or a block instead of your content, that's your answer regardless of what robots.txt permits.
- Check your CDN's bot-management dashboard directly — Cloudflare's AI Crawl Control, Fastly's bot rules, or your WAF's managed rule sets — rather than assuming robots.txt is the only control in play. These dashboards are usually where the real decision gets made.
- If you're on Cloudflare, revisit your settings before any policy deadline changes defaults — new and free-tier accounts are increasingly defaulted toward blocking, not allowing, so a setting you never touched can still change under you.
- Re-test after every CDN, hosting, or security-plugin change. A single new WAF rule or "bot protection" toggle can silently undo months of GEO work without ever touching your robots.txt file.
For a bilingual site, this matters twice over: an English-language WAF rule set or CDN configuration applies identically to your Chinese-language pages, so a single misconfigured rule can make your entire bilingual GEO investment invisible to AI crawlers in both languages at once.