Cloudflare: 416 billion AI bot requests blocked since July
Google sees 3.2x more webpages than OpenAI. Cloudflare CEO Matthew Prince wants Google to separate search crawling from AI crawling.
Cloudflare has blocked 416 billion AI bot requests for customers since July 1, according to CEO Matthew Prince. To highlight Google’s massive edge in AI, Prince revealed that Google can see 3.2x more pages on the web than OpenAI.
Why we care. Generative AI systems are consuming huge amounts of web content, and many publishers have no way to push back. Cloudflare’s numbers show how aggressively AI bots are scraping the web – and how uneven access is for the big AI companies.
Driving the news. Cloudflare customers have been automatically blocking AI crawlers since the company launched its pay-per-crawl initiative July 1, Prince said yesterday at WIRED’s Big Interview event. Since then, Cloudflare has logged 416 billion blocked AI bot requests.
Cloudflare’s analysis showed that Google sees:
- 3.2x more webpages than OpenAI.
- 4.6x more than Microsoft.
- 4.8x more than Anthropic or Meta.
- “They have this incredibly privileged access,” Prince said.
The big picture. Google gives publishers a lose-lose choice: Block AI training and risk vanishing from Google Search, or allow Googlebot and accept AI scraping.
- “You can’t opt out of one without opting out of both, which is crazy. You shouldn’t get to use your monopoly of yesterday to secure a monopoly of tomorrow,” Prince said.
The AI picture. According to Prince:
- AI represents a major platform shift that could reshape the web’s business model.
- Cloudflare wants to prevent consolidation, keep the web open, and help creators and businesses navigate the transition.
- Publishers already blocking AI crawlers are seeing encouraging results, Prince added.
What’s next. The value of “creative, original human thought” will rise as AI models chase higher-quality training data, opening doors to paid licensing, Prince said. Meanwhile, Cloudflare is pushing AI giants – especially Google – to separate search crawling from AI crawling. Prince said:
- “Google is the problem here. It is the company that is keeping us from going forward on the internet, and until we force them – or hopefully convince them – that they should play by the same rules as everyone else and split their crawlers up between search and AI, I think we’re going to have a hard time completely locking all the content down.”
The story. Cloudflare Has Blocked 416 Billion AI Bot Requests Since July 1 (subscription required)
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
About the Author
