Cloudflare changes default rules for AI agent crawlers

Cloudflare is changing how AI agent crawlers are handled on parts of the web. Starting September 15, some adsupported pages will block agent and training crawlers by default, while search crawlers will still be allowed. The company announced the update on July 1 and said the new controls are already available to customers, including freetier users. Cloudflare has split crawler activity into three categories: search, agent, and training. Search is for indexing pages to answer later, agent refers to systems acting in real time for a user, and training covers crawlers that feed data into model weights. The article says publishers can keep the default blocks or opt out through security settings before the change takes effect. It also notes that businesses running AI agents may need to negotiate access rather than rely on simple useragent changes, because Cloudflare’s blocking works at the network level. The piece highlights that the update could also affect mixeduse crawlers, including Googlebot, if sites choose stricter settings.