robots.txt for AI Crawlers

Your robots.txt file tells crawlers what they may fetch. Several AI companies run more than one crawler, each with a different job, so a single "block all bots" rule can quietly remove you from AI search. This guide explains the main ones and shows sample rules.

Where the file lives

robots.txt must sit at the root of your domain: https://yourdomain.com/robots.txt. It is a request that well-behaved crawlers follow. It is not a security tool, since it does not stop anyone who ignores it. The standard is described in RFC 9309 (RFC 9309).

The main AI-related user agents

The table lists agents as documented by their vendors at the time of writing. Vendors change these, so confirm on the linked pages before you rely on them.

User agent	Operator	Purpose as documented
`GPTBot`	OpenAI	Collects content that may be used to train models. Docs
`OAI-SearchBot`	OpenAI	Surfaces sites in ChatGPT search results. Same docs
`ChatGPT-User`	OpenAI	Fetches pages when a user's request calls for it. Same docs
`ClaudeBot`	Anthropic	Training data collection. Help Center
`Claude-SearchBot`	Anthropic	Search result quality. Same source
`Claude-User`	Anthropic	User-directed fetches. Same source
`PerplexityBot`	Perplexity	Surfaces and links sites in Perplexity results. Docs
`Perplexity-User`	Perplexity	User-initiated fetches. Same docs
`Google-Extended`	Google	A token that controls use of crawled content for Gemini training and grounding; does not affect Search. Docs
`Googlebot`	Google	Crawls for Google Search, including AI features in Search
`Bingbot`	Microsoft	Crawls for Bing
`Applebot-Extended`	Apple	A token controlling use of Applebot-crawled content for Apple AI training
`CCBot`	Common Crawl	Builds an open web archive that many AI projects use

The most common mistake

Many sites paste in a list that blocks every AI-related name. That blocks the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) along with the training ones. The business then asks why it never appears in AI answers. Search crawlers and training crawlers are separate on purpose. Decide about each group separately.

Sample 1: allow everything

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This is the simplest setup and the right one if you want maximum visibility and do not mind training use.

Sample 2: allow AI search, block training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This asks training-related crawlers to stay out while leaving search crawlers like OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, and Bingbot under the general allow. Note that Google-Extended also covers grounding in Gemini apps and the Gemini API per Google's documentation, so you are choosing to limit that use too. Whether any given vendor honors a rule exactly as described is up to the vendor; check their documents.

Sample 3: block a private area

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Allow: /

How rules are matched

A crawler follows the most specific User-agent group that matches its name. If a group names GPTBot, GPTBot ignores the * group.
Paths are case-sensitive. /Admin/ and /admin/ are different.
An empty Disallow: means allow everything.
Comments start with #.

Other things that block bots

robots.txt is not the only gate. Check for:

Firewall or CDN bot rules. Some security settings challenge or block unknown crawlers regardless of robots.txt.
Meta robots tags. noindex on pages keeps them out of search indexes.
Login walls and heavy JavaScript. A crawler that cannot see your text cannot cite it.
Slow servers. Timeouts can look like blocks.

How to test your file

Open your robots.txt in a browser and read it top to bottom.
Search it for each name in the table. Note any group that disallows /.
Check for a final User-agent: * group with Disallow: /, which blocks everyone not named earlier.
Test a few URLs in Google Search Console's robots.txt report for Googlebot. For other agents, read the rules by hand.

SearchSignal's paid report checks your robots.txt against these agents and flags any that are blocked, as part of an on-site audit. A free standalone checker is planned.

Keep it current

Review the file a few times a year. Agent names and purposes change, and some hosting platforms and plugins edit robots.txt without telling you. See also how to get mentioned in ChatGPT.

Common questions

Does robots.txt remove me from AI answers that already exist?

No. It controls future fetching. Content already collected or indexed may remain according to each operator's own policies.

Do all AI companies obey robots.txt?

Reputable operators say they do, with some exceptions for user-initiated fetches. It is a convention, not a lock. For sensitive pages use authentication.

Should I list every crawler by name?

Only the ones you want to treat differently from the default. Fewer rules are easier to keep correct.

What about a crawler not on this list?

New agents appear often. Check each vendor's documentation, and review your server logs for unfamiliar user agents.

Published 2026-10-30. We cite vendor documentation where we state facts about it; vendors change products often, so check the linked pages for current details. This guide describes what you can influence and does not promise any result.