AI Crawl Control

AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors. Content Format insights The Metrics tab…

Cloudflare's network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages. When enabled via AI Crawl Control > Quick Actions, AI training crawlers that request a page with a canonical tag pointing…

AI Crawl Control metrics have been enhanced with new views, improved filtering, and better data visualization. Path pattern grouping In the Metrics tab > Most popular paths table, use the new Patterns tab that groups requests by URI pattern (/blog/*,…

New reference documentation is now available for AI Crawl Control: GraphQL API reference — Query examples for crawler requests, top paths, referral traffic, and data transfer. Includes key filters for detection IDs, user agents, and referrer domains. Bot reference —…

Account administrators can now assign the AI Crawl Control Read Only role to provide read-only access to AI Crawl Control at the domain level. Users with this role can view the Overview, Crawlers, Metrics, Robots.txt, and Settings tabs but cannot…

The Overview tab is now the default view in AI Crawl Control. The previous default view with controls for individual AI crawlers is available in the Crawlers tab. What's new Executive summary — Monitor total requests, volume change, most common…