AI Crawlers on Your Website: Who to Let In, Who to Block

While marketers deal with visibility in AI answers, website administrators deal with the other end of the same phenomenon: AI bots are downloading content massively. According to Cloudflare data, the share of GPTBot (OpenAI) in verified bot traffic grew from 4.7% in the summer of 2024 to 11.7% a year later; its activity grew by hundreds of percent year on year, and ClaudeBot (Anthropic) holds just under 10%. The question is no longer whether AI bots visit you – but what to do about it.

What you get back for the crawl

Cloudflare introduced a useful metric: the crawl-to-referral ratio – how many pages a bot downloads per one visit its service sends back to the website. The 2025 numbers speak clearly:

• Google: roughly 14 downloaded pages per 1 referred visit.
• Bing: roughly 41:1.
• Perplexity: in the order of hundreds to one – the best of the pure AI companies.
• OpenAI: in the order of a thousand to one.
• Anthropic: in the order of tens of thousands to one.

No wonder websites defend themselves: after Cloudflare made AI crawler blocking the default for new domains in July 2025, the number of sites blocking AI training exceeded 2.5 million.

Block or allow? It depends who you are

For a publisher living off content visits, blanket blocking can be defensible. For a company, e-shop or service it is usually an own goal: whoever blocks AI bots will not get into the answers that today bring the highest-quality traffic. The key is to distinguish bot types:

• Training bots (e.g. GPTBot, ClaudeBot) – collect data for model training. Blocking = a lower long-term chance the model “knows” you.
• Search bots (e.g. OAI-SearchBot, PerplexityBot) – serve AI search with citations. Blocking = an immediate loss of citations and referrals.
• On-demand bots (e.g. ChatGPT-User) – fetch a page when a user needs it in a chat right now. Blocking = broken links to you directly in conversations.

Practical recommendations

1. Marketing website / e-shop: let AI bots through – visibility in answers is worth more than saved server capacity. Manage individual user agents in robots.txt, not a blanket ban.
2. Paid / unique content: consider blocking training bots, but keep search and on-demand bots – otherwise you disappear from citations.
3. Help bots get to the point: an llms.txt file and a clean site structure shorten the path to what the model should know about you.
4. Watch your logs. The share of AI bots in access logs is now as important a metric as traffic from Google.

And remember: robots.txt only decides about your own website. Whether models recommend you is largely decided outside it – in mentions and articles on sites AI bots crawl with top priority. That is exactly where PR articles and mentions from the Linketica marketplace get you.