This tool can protect your website from bots scraping your data for AI

Cloudflare has unveiled AI Audit, a new suite of tools designed to give website owners and bloggers more control over their content and how it’s used by automated systems. The security tool allows users to block or allow artificial intelligence (AI) bots that frequently crawl websites to scrape data. The company is offering the tool to all of its existing customers at no additional cost. Cloudflare is also adding a feature that will allow website owners to see detailed analytics into which bots visit the platform and their behavioral patterns.

With the increasing popularity of generative AI technology, there has been a rush to train large language models (LLMs) with human-created data. This is important not only to create a base model but also to improve it. However, the main problem here is access to publicly available data sources. Most AI companies have already gone through these datasets and now need more data to train AI models.

AI bots are designed to help developers gather more information to train AI models. They can be thought of as systematic programs that imitate a real user who can open websites and copy text, image, and video data. These AI bots can sift through large amounts of data in a short time and feed it to the AI ​​model. In recent times, several media companies and major websites have filed lawsuits against AI companies for plagiarism and illegally using data to feed LLMs.

Cloudflare’s AI Audit tool is a protective layer that can prevent such bots from accessing websites, the company revealed in a press release which has also made improvements to the tool to give users more control over which bots are restricted and which bots are given access. This is useful in cases where the platform has a deal with an AI company and doesn’t mind their bots using the data. Alternatively, the website owner may want to give access to certain AI models that attribute the source of the data to get better reach.

Cloudflare emphasized that it is also building a workflow that will allow website owners to set a fair price for their content. AI bot owners, on the other hand, can make transactions with this firewall, and once the fee is paid, they are granted the right to scan content. The company emphasized that its marketplace-like tool could be useful for users who don’t have the bandwidth or resources to negotiate and close deals with every AI company that approaches their website.