Cloudflare, an internet infrastructure firm, is rolling out a new 'game-changing' system that allows millions of websites, including Sky News and Buzzfeed, to block AI bots from accessing their content without permission. This initiative aims to protect copyright and enable publishers to eventually charge AI firms for content scraping, addressing widespread concerns from creators who accuse AI companies of unauthorized use of their work. The system is currently active on a million websites, and Cloudflare is developing a 'Pay Per Crawl' system to ensure fair compensation for content creators.
Millions of websites to get 'game-changing' AI bot blocker
Artificial intelligence
AI Summary
TL;DR: Key points with love ❤️Cloudflare, an internet infrastructure firm, is rolling out a new 'game-changing' system that allows millions of websites, including Sky News and Buzzfeed, to block AI bots from accessing their content without permission. This initiative aims to protect copyright and enable publishers to eventually charge AI firms for content scraping, addressing widespread concerns from creators who accuse AI companies of unauthorized use of their work. The system is currently active on a million websites, and Cloudflare is developing a 'Pay Per Crawl' system to ensure fair compensation for content creators.
Trending- 1 March (2025): Cloudflare reported that AI Crawlers generate more than 50 billion requests to its network every day.
- 2 Recent: Cloudflare began rolling out its new system to block AI bots from accessing website content without permission.
- 3 Future: Cloudflare is developing a 'Pay Per Crawl' system to allow content creators to request payment from AI companies.
- Millions of websites gain the ability to block AI bots
- Potential for a new economic model where content creators can charge AI companies for access
- Increased control for publishers over their content's use by AI
- Ongoing legal and legislative battles over AI and copyright on both sides of the Atlantic
What: Cloudflare is rolling out a new system to allow millions of websites to block AI bots (crawlers) from accessing their content without permission. This system aims to protect copyright and enable publishers to eventually charge AI firms for content scraping. A 'Pay Per Crawl' system is under development. This addresses concerns from writers, artists, musicians, and actors about AI firms training systems on their work without permission or payment.
When: Recent rollout (article published July 1, 2025); March (Cloudflare reported AI crawler activity); Future (Pay Per Crawl system development).
Where: Global internet (Cloudflare hosts around a fifth of the internet); UK (legislative battle over copyright and AI).
Why: To protect content creators' copyright, ensure a fair value exchange on the internet, support quality journalism, hold AI companies accountable, and prevent AI crawlers from generating answers without directing visitors to original sources, thereby depriving creators of revenue.
How: Cloudflare's technology targets AI firm bots/crawlers, initially applying by default to new users and sites that participated in an earlier blocking effort. The company is also developing a 'Pay Per Crawl' system to allow content creators to request payment from AI companies.