AI Crawl Control

Account administrators can now assign the AI Crawl Control Read Only role to provide read-only access to AI Crawl Control at the domain level. Users with this role can view the Overview, Crawlers, Metrics, Robots.txt, and Settings tabs but cannot…

The Overview tab is now the default view in AI Crawl Control. The previous default view with controls for individual AI crawlers is available in the Crawlers tab. What's new Executive summary — Monitor total requests, volume change, most common…

Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control. For AI crawler operators Discovery API A new authenticated API endpoint allows verified crawlers to…

AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules, Redirect Rules, and robots.txt files.…

AI Crawl Control now includes a Robots.txt tab that provides insights into how AI crawlers interact with your robots.txt files. What's new The Robots.txt tab allows you to: Monitor the health status of robots.txt files across all your hostnames, including…

AI Crawl Control now provides enhanced metrics and CSV data exports to help you better understand AI crawler activity across your sites. What's new Track crawler requests over time Visualize crawler activity patterns over time, and group data by different…