AI Crawl Control

AI Crawl Control metrics have been enhanced with new views, improved filtering, and better data visualization. Path pattern grouping In the Metrics tab > Most popular paths table, use the new Patterns tab that groups requests by URI pattern (/blog/*,…

New reference documentation is now available for AI Crawl Control: GraphQL API reference — Query examples for crawler requests, top paths, referral traffic, and data transfer. Includes key filters for detection IDs, user agents, and referrer domains. Bot reference —…

Account administrators can now assign the AI Crawl Control Read Only role to provide read-only access to AI Crawl Control at the domain level. Users with this role can view the Overview, Crawlers, Metrics, Robots.txt, and Settings tabs but cannot…

The Overview tab is now the default view in AI Crawl Control. The previous default view with controls for individual AI crawlers is available in the Crawlers tab. What's new Executive summary — Monitor total requests, volume change, most common…

Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control. For AI crawler operators Discovery API A new authenticated API endpoint allows verified crawlers to…

AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules, Redirect Rules, and robots.txt files.…