AI Crawl Control now includes a Robots.txt tab that provides insights into how AI crawlers interact with your robots.txt files.
What’s new
The Robots.txt tab allows you to:
- Monitor the health status of
robots.txtfiles across all your hostnames, including HTTP status codes, and identify hostnames that need arobots.txtfile. - Track the total number of requests to each
robots.txtfile, with breakdowns of successful versus unsuccessful requests. - Check whether your
robots.txtfiles contain Content Signals directives for AI training, search, and AI input. - Identify crawlers that request paths explicitly disallowed by your
robots.txtdirectives, including the crawler name, operator, violated path, specific directive, and violation count. - Filter
robots.txtrequest data by crawler, operator, category, and custom time ranges.
Take action
When you identify non-compliant crawlers, you can:
- Block the crawler in the Crawlers tab
- Create custom WAF rules for path-specific security
- Use Redirect Rules to guide crawlers to appropriate areas of your site
To get started, go to AI Crawl Control > Robots.txt in the Cloudflare dashboard. Learn more in the Track robots.txt documentation.
Source: Cloudflare
Latest Posts
- MC1557760: Microsoft Teams Increases Registration Capacity for Town Halls and Webinars to 20,000

- MC1557764: Agent 365 Business Analytics Adds Access Management for Admin Role and Permission Control

- MC1557754: Microsoft Copilot Adds Usage-Based Billing to Continue Model Access with Copilot Credits

- MC1557766: Microsoft Viva Adds AI-Powered Interactive Insights to Agent 365 Business Analytics







