How AI crawlers affect your website’s visibility

Letting AI crawlers in makes your content eligible for citation in AI answers. Here’s what they do, when it makes sense to restrict them, and why where you block matters most.

Dark concrete interior with a glowing teal doorway casting a sharp beam of light across the floor.

AI crawlers are the bots that companies such as OpenAI, Anthropic, Google and Perplexity send to read websites. Some collect content to train their models. Others index pages or fetch them in real time, so tools like ChatGPT Search, Perplexity and Google AI Overviews can find and cite them when someone asks a question.

For most organisations, letting those search and retrieval crawlers in is the right call. AI tools cite their sources with a link back, so allowing them makes your content eligible to appear in AI answers. Block them, and those tools still answer questions about you. They just use directories, reviews and competitors’ comparison pages instead of your own words.

There are good reasons to restrict some access. Publishers may block training while keeping retrieval open, and sites with legal or medical guidance may restrict both on those sections. But where you block matters more than whether you block.

Cloudflare’s ‘Block’ settings for AI training now also apply to Googlebot, and your hosting control panel may block crawlers separately, whatever your CDN says. Choose the wrong setting, and your pages can quietly drop out of Google while every dashboard still looks fine.

Our AI Technical Lead, Mike Onslow, has written a full guide on whether to block or allow AI crawlers. It covers which crawlers do what, how to set rules in robots.txt and Cloudflare, and how to check you haven’t blocked Google by mistake.



Looking for something specific?