What you need to know!
Instructions
- Start by Fetching Agents
Access the latest list of web bots and agents by simply loading the tool. The list is fetched from a continuously updated source ensuring you have the most current data. - Filtering Options
Use the category dropdown to filter bots by their type, such as "Scraper" or "SEO Crawler". For a more specific search, use the search bar to find bots by name. - Selecting Agents
Click the "Add" button next to any bot's name to select it for your custom rule set. This bot will move to your "Selected Bots" list. - Batch Operations
Utilize the "Add All" button to select all bots currently visible based on your filter or search criteria. - Generate Rules
With bots selected, choose to generate either a `robots.txt` file or a Cloudflare WAF ruleset tailored to block the selected agents. - Copy to Clipboard
Once generated, use the "Copy to Clipboard" button to easily copy the ruleset, ready for implementation on your site or Cloudflare configuration.
Note about Robots.txt Usage
The `robots.txt` file is a powerful tool for managing web crawler access to your site. However, compliance is voluntary, and malicious bots may ignore these directives. Use this generated `robots.txt` as a first line of defense, and consider additional security measures for comprehensive protection. More info on robots.txt
Note about Cloudflare WAF and Character Limit
When generating Cloudflare WAF rules, be mindful of the 4096 character limit per rule. Our tool helps streamline your ruleset by consolidating bot identifiers, but complex selections may exceed this limit. If necessary, split extensive rules into multiple segments to ensure compatibility with Cloudflare's constraints. Cloudflare WAF Rules
AI Scrapers VS SEO Crawlers
We conducted an analysis on our website to determine the proportion of bots, specifically aibots, contributing to our traffic:
- AI Data Scrapers (orange)
- AI Search Crawlers (yellow)
- SEO Crawlers (pink)
- Search Engine Crawlers (blue)
Upon each new content upload, aiscrapers consistently outpaced seo crawlers in reaching our site.
Useful Resources
- Known Agents (Dark Visitors)
- user-agents.net
- Blocking Known AI bots using Cloudflare
- Further reading: AI Bots & Cloudflare
- Cloudflare WAF Rule V3
- AI robots.txt, AI scrapers block ai scrapers
- Content Signals - Introducing Content Signals, Cloudflare's implementation of a mechanism for allowing website publishers to declare how automated systems should use their content.
Disclaimer
This tool is provided "as-is" without warranty of any kind. By using this tool, you acknowledge the inherent risks and agree to assume all responsibility for the outcomes of implementing the generated `robots.txt` or Cloudflare WAF rulesets. We encourage reviewing each rule for relevance and accuracy before application to avoid unintended access restrictions or security implications. Use at your own risk.
The updated list of Agent Bot Information is retrieved and collated from: Known Agents. Thank you to Gavin King for keeping this project alive.