If you want your business to show up in AI answers, you need to allow the major AI crawlers — GPTBot, ClaudeBot, Google-Extended, and PerplexityBot — in your robots.txt file. But allowing them there isn’t the whole job. A growing number of sites unknowingly block these bots at the network level, because platforms like Cloudflare now block AI crawlers by default. Here’s how to check, and how to fix it.
Table of Contents
Why AI Crawler Access Matters
AI answer engines like ChatGPT, Perplexity, and Google’s AI Overviews don’t invent their answers from nothing. When someone asks them a question, they read pages across the web and compose a response from what they find — citing the sources they used. To be one of those sources, your site has to be readable by the crawlers those engines send out.
If a crawler is blocked, none of your content matters. It doesn’t get read, it doesn’t get cited, and the competitor whose site is open gets named instead. This is the quietest way to be invisible in AI search — nothing looks broken, you just never show up.
The AI Crawlers That Matter
These are the crawlers worth knowing about — who runs them and what they feed:
| Crawler | Run by | What it feeds |
|---|---|---|
| GPTBot | OpenAI | ChatGPT — training and search |
| OAI-SearchBot | OpenAI | ChatGPT’s live search results |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to browse it |
| ClaudeBot | Anthropic | Claude |
| Google-Extended | Gemini and Google’s AI models | |
| PerplexityBot | Perplexity | Perplexity’s answers and citations |
| Applebot-Extended | Apple | Apple Intelligence |
| CCBot | Common Crawl | An open dataset many AI models train on |
How to Allow AI Crawlers in robots.txt
Your robots.txt file lives at the root of your site (yoursite.com/robots.txt) and tells crawlers what they can and can’t access. To explicitly welcome the AI crawlers, add rules like these:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /On WordPress, you can usually manage this through your SEO plugin rather than editing a file by hand. The key thing to check: make sure there’s no rule that says Disallow: / under any of these bots — that single line blocks the crawler from your entire site.
The Cloudflare Trap
Here’s where most people get caught. Even with a perfect robots.txt, your site can still block AI crawlers at the network level — before the request ever reaches your content. Cloudflare, which sits in front of a huge share of websites, now blocks AI crawlers by default on many plans. It does this to protect site owners from having their content scraped, which is a reasonable default for some businesses. But if your goal is to be found by AI, that default is working against you.
If you use Cloudflare, check its dashboard for AI bot or “block AI scrapers” settings and make sure the crawlers you want are allowed there too. Both layers — your robots.txt and your network settings — have to agree. One says yes and the other says no, and no wins.
The Caching Gotcha We Ran Into Ourselves
One more thing, because we hit it on our own site: robots.txt gets cached hard. We updated ours to allow every AI crawler, confirmed the correct rules were saved — and the live file still showed the old block. The fix wasn’t another edit; it was clearing the cache. Your host, your CDN, and your browser can all serve a stale copy. So after any robots.txt change, purge every cache layer and re-check the live file in a private browser window before you assume it worked. If the live file still shows the block, it’s cached, not saved wrong.
Should You Allow All of Them?
This is a genuine choice, not a rule. If your goal is visibility — you want AI engines to find, understand, and recommend your business — then yes, allow them. For most local and service businesses, being cited by AI is pure upside.
Some businesses block AI crawlers on purpose, to keep their content out of training data. That’s a legitimate position, especially for publishers whose content is the product. Just know the trade-off: block the crawlers and you also give up the chance to be the answer an AI gives. One clarification worth knowing — blocking Google-Extended stops Google from using your content to train Gemini, but it does not remove you from Google’s AI Overviews, which rely on the regular Googlebot. The two are separate.
How to Confirm It’s Working
After you make changes, verify two things. First, load yoursite.com/robots.txt in a private browser window and confirm the AI crawlers show Allow: /, not Disallow: /. Second, over the following days, check your server access logs for the crawler names in the table above. Seeing GPTBot or PerplexityBot appear in your logs is your proof they can now reach you.
Allowing AI crawlers is one piece of a larger discipline called Generative Engine Optimization — making your whole site readable, trustworthy, and citable to AI systems. If you’d rather have it handled for you, that’s exactly what our GEO services do.
Frequently Asked Questions
Do I have to allow AI crawlers?
No — it’s a choice. If you want your business to appear in AI answers, you should allow them. If you’d rather keep your content out of AI systems entirely, you can block them. Most businesses that want more customers benefit from allowing access.
Does blocking Google-Extended remove me from Google’s AI Overviews?
No. Google-Extended only controls whether your content is used to train Google’s AI models like Gemini. AI Overviews in search rely on the standard Googlebot, so blocking Google-Extended does not take you out of AI Overviews.
Why is Cloudflare blocking AI bots?
Cloudflare added AI-crawler blocking to help site owners control whether their content gets scraped for AI training, and made it a default on many plans. It’s protective by design — but if your goal is to be found by AI, you’ll want to change that setting so the crawlers can reach you.
How do I know if a crawler is actually reaching my site?
Check your server access logs for the crawler’s name — GPTBot, ClaudeBot, PerplexityBot, and so on. If those names appear in your logs after you’ve allowed them, they’re successfully reaching your pages.
Will allowing AI crawlers slow down my site?
Not meaningfully. AI crawlers visit at a modest rate, similar to traditional search crawlers. A well-hosted, performance-tuned site handles them without any noticeable impact on speed for your human visitors.