AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules, Redirect Rules, and robots.txt files.
What’s new
Status code distribution chart
The Metrics tab includes a status code distribution chart showing HTTP response codes (2xx, 3xx, 4xx, 5xx) over time. Filter by individual crawler, category, operator, or time range to analyze how specific crawlers interact with your site.
Extended actions menu
Each crawler row includes a three-dot menu with per-crawler actions:
- View Metrics — Filter the AI Crawl Control Metrics page to the selected crawler.
- View on Cloudflare Radar — Access verified crawler details on Cloudflare Radar.
- Copy User Agent — Copy user agent strings for use in WAF custom rules, Redirect Rules, or robots.txt files.
- View in Security Analytics — Filter Security Analytics by detection IDs (Bot Management customers).
- Copy Detection ID — Copy detection IDs for use in WAF custom rules (Bot Management customers).
Get started
- Log in to the Cloudflare dashboard, and select your account and domain.
- Go to AI Crawl Control > Metrics to access the status code distribution chart.
- Go to AI Crawl Control > Crawlers and select the three-dot menu for any crawler to access per-crawler actions.
- Select multiple crawlers to use bulk copy buttons for user agents or detection IDs.
Learn more about AI Crawl Control.
Source: Cloudflare
Latest Posts
- Dynamics 365 Contact Center – Automatically handle waiting conversations when queues close or become unstaffed [MC1323827]
- Dynamics 365 Project Operations – Enable Project Revenue recognition using cost estimate instead of EAC [MC1323835]
- Updates available for Microsoft 365 Apps for Current Channel [MC1323310]
- Data Privacy: Microsoft Online Services Subprocessor Disclosure [MC1323272]
