Cloudflare announced on Monday that it will create a new marketplace where website owners can sell access to their site’s content for AI bots. This move is part of Cloudflare CEO Matthew Prince’s plan to give publishers more control over how and when AI models scrape their sites.
“If creators aren’t paid, they’ll stop creating, and that needs to change,” Prince says.
To support this plan, Cloudflare has introduced a free tool called AI Audit. This tool gives website owners a dashboard to see how often AI models crawl their sites and why. With just one click, they can block AI bots. Owners can choose to block all scrapers or allow specific ones if they find them useful.
A demo of AI Audit showed how it tracks scrapers from different AI providers like OpenAI and Meta. Cloudflare aims to solve a key problem for smaller publishers: how can they survive if people use AI tools like ChatGPT instead of visiting their sites? Many small websites are scraped daily for information to train AI models without receiving any compensation, which could hurt their traffic and income.
Earlier this summer, AI startup Perplexity was accused of scraping sites that didn’t want to be crawled. In response, Cloudflare added a button to help customers block all AI bots easily. Many website owners reported that excessive scraping felt like a DDoS attack, slowing down their servers and increasing costs.
What if a site wants to block Perplexity but not OpenAI? Prince says customers want tools that let them choose which AI models can access their content. Cloudflare’s new features will allow site owners to block some crawlers while allowing others.
Even big publishers with licensing deals, like TIME and The Atlantic, don’t have a clear view of how much ChatGPT scrapes their sites. They often have to rely on OpenAI’s reports, which can affect their licensing agreements.
Cloudflare’s upcoming marketplace, set to launch next year, will help smaller publishers negotiate with AI providers too. “Let’s give everyone the chance to do what only big publishers have done before,” Prince said. “What if you could set a price for your content?”
While the details of the marketplace are still unclear, Prince suggests that websites might charge based on how often their content is scraped or ask for credit from AI labs. He believes this change will be good for the AI ecosystem, as the current model where AI companies use content for free isn’t sustainable.
Conclusion
Cloudflare’s new marketplace could change how AI bots access website content, offering more control and potential revenue for smaller publishers. By introducing tools like AI Audit, Cloudflare aims to protect creators and ensure that everyone can benefit from the growing AI landscape.
For more tech, AI, cyber security, and digital marketing insights, visit Daily Digital Grind. If you’re interested in contributing, check out our Write for Us page to submit your guest posts!