Robots.txt Validator
Check robots.txt syntax and test URL access rules
18 known AI bots are not blocked.
使用方法
Paste your robots.txt content into the editor. The validator checks every line for syntax errors, unknown directives, and AI bot rules. Use the URL tester to check whether a specific path is allowed or blocked for any user-agent.
示例
Checking a WordPress robots.txt
Paste your WordPress robots.txt to verify that /wp-admin/ and /wp-includes/ are properly blocked while public content remains accessible. The validator flags any syntax issues and confirms your sitemap URL is valid.
Auditing AI bot rules
The validator detects rules targeting AI crawlers like GPTBot, ClaudeBot, and Google-Extended. See exactly which AI bots are blocked and which can still access your content.
Testing a specific URL path
Enter a URL path like /api/internal and select a user-agent to check whether it is allowed or disallowed. The tool shows which rule matched and on which line.
Frequently asked questions
What syntax does robots.txt use?
Robots.txt uses simple directives: User-agent (which bot), Allow and Disallow (which paths), Sitemap (where to find your sitemap), and Crawl-delay (seconds between requests). Each rule group starts with a User-agent line followed by one or more Allow/Disallow lines.
How does the URL tester determine if a path is allowed?
It follows the Google specification: find the most specific matching rule (longest path pattern) for the selected user-agent. If no specific user-agent match exists, it falls back to the wildcard (*) rules. No matching rules means the URL is allowed.
Which AI bots does the validator detect?
It detects GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, PerplexityBot, Meta-ExternalAgent, and other known AI crawlers. The validator shows which bots have Disallow rules blocking them.
Does Google respect Crawl-delay?
No. Google ignores the Crawl-delay directive. Use Google Search Console to control Googlebot crawl rate. Bing and Yandex do respect Crawl-delay.
Is robots.txt case-sensitive?
Directive names (User-agent, Disallow) are case-insensitive. URL paths in Allow and Disallow rules are case-sensitive. /Admin/ and /admin/ are treated as different paths by most crawlers.
About this tool
Validate your robots.txt file for syntax errors, AI bot rules, and crawl directives. Test if specific URLs are allowed or blocked for any user-agent.
所有计算均在您的浏览器本地执行。您的数据不会离开您的设备。