Robots.txt tester for search and AI crawlers
Free robots.txt tester. Paste your file, test any URL, and see whether Googlebot, GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers are allowed.
Paste the file from https://yoursite.com/robots.txt. Matching follows RFC 9309 as Google applies it: the most specific group for the crawler wins, then the longest matching rule wins, and Allow wins a tie. * and $ wildcards are supported.
| Crawler | Operator | Status | Rule that decided it |
|---|
Blocking AI crawlers without blocking search
Most AI companies now use separate crawlers for model training and for live answers. Blocking GPTBot stops OpenAI training crawls but not OAI-SearchBot, which powers ChatGPT search results. Google-Extended controls Gemini training use, but Google Search still crawls with Googlebot. Decide per crawler, then test the result here.
robots.txt is a request, not access control. Well-behaved crawlers follow it. Use authentication for anything that must stay private.
Questions people ask
Add a group for User-agent: GPTBot with Disallow: / and leave OAI-SearchBot allowed. GPTBot is used for training; OAI-SearchBot is used for ChatGPT search results.
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use a noindex meta tag on a crawlable page to remove it.
The longest matching path wins. If they are the same length, Allow wins. This tool applies the same logic.
No. Parsing and matching run in your browser.
Let your AI agent check every page, not one at a time.
SearchSignal crawls your site, ranks fixes by impact, and hands the list to Cursor, Claude, ChatGPT, or any HTTP agent. Free plan, no card. Pro adds licensed keyword and backlink data for $19.99/mo.