Robots.txt and AI crawlers: how it works and what to allow

A few years ago, most site owners only thought about robots.txt in terms of search engines. Now a growing number of crawlers collect web content for AI products: some to train models, some to power AI search answers, and some to fetch a page when a user asks an assistant a question. Each operator documents its own user agents and what they are used for, and robots.txt is the main way to express your preferences to them.

In this article you will learn:

• what robots.txt does and, just as importantly, what it does not do
• the syntax, with examples
• how to test your file
• the AI crawler user agents that operators document, with what each is used for, according to their own documentation at the time of writing
• the trade-offs of blocking or allowing AI crawlers

What robots.txt is (and is not)

Robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may request. The format is known as the Robots Exclusion Protocol, published as an IETF standard (RFC 9309).

Google's Search Central documentation describes its main purpose as managing crawler traffic, mainly to avoid overloading your site with requests. The same documentation makes two points worth remembering:

• It is not a way to keep a page out of search results. Google states that robots.txt is not a mechanism for keeping a web page out of Google. A blocked URL can still appear in results if other pages link to it, although Google will not have crawled its content. To keep a page out of the index, Google recommends noindex or password protection instead.
• Not every crawler obeys it. Google notes that other crawlers might not follow the rules. Robots.txt is a request, not access control. Well-behaved crawlers from reputable operators follow it; scrapers and bad actors can ignore it.

If content must not be accessed, put it behind authentication. Robots.txt is also public, so do not use it to list secret URLs; anyone can read the file.

Syntax basics

The file must be named robots.txt, sit at the root of the host (for example https://www.example.com/robots.txt) and be UTF-8 encoded, according to Google's documentation. A robots.txt file applies only to the host and protocol it is served on, so a subdomain needs its own file.

The file is made of groups. Each group starts with one or more User-agent lines followed by rules:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public-help/

Sitemap: https://www.example.com/sitemap.xml

• User-agent names the crawler the group applies to. * means any crawler not matched by a more specific group.
• Disallow gives a path the crawler should not request.
• Allow permits a path inside a disallowed section.
• Sitemap gives the full URL of your XML sitemap. It is not tied to a group.

Some details from Google's documentation, at the time of writing:

• Paths are case sensitive. Disallow: /file.asp does not block /FILE.asp.
• Wildcards: * matches any sequence of characters and $ marks the end of the URL. For example, Disallow: /*.pdf$ blocks URLs ending in .pdf.
• Group selection: a crawler follows the group with the most specific user agent that matches it, and ignores the other groups. This catches people out: if you create a group for GPTBot, that crawler will follow only that group, not the rules under User-agent: *.
• Conflicting rules: Google uses the most specific rule (the longest matching path), and when rules conflict equally, the least restrictive one.
• File size: Google enforces a limit of 500 KiB; content after that is ignored.
• Errors: Google treats most 4xx responses (except 429) as if no robots.txt exists, meaning everything may be crawled. A 5xx server error causes Google to pause crawling and later fall back to the last good version.

Other crawlers may interpret edge cases differently, so keep your file simple.

AI crawler user agents documented by operators

The table below lists user agents that we could verify in each operator's own documentation at the time of writing. Operators add, rename and redefine crawlers, so check the cited documentation before relying on this list.

Notice the pattern: several operators now separate crawlers by purpose. Training crawlers (such as GPTBot, ClaudeBot and, as a control token, Google-Extended) are distinct from search crawlers (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) and from user-initiated fetchers (such as ChatGPT-User, Claude-User and Perplexity-User). This means you can express different preferences for each purpose.

Common Crawl's data is openly available and widely used for research and other purposes, so blocking CCBot affects more than one downstream use. Common Crawl also warns that some crawlers falsely identify themselves as CCBot, which applies to any user agent: a request claiming to be a named crawler is not proof. Some operators, including Anthropic, publish IP address information that can be used to verify genuine requests.

Example configurations

These are illustrative examples. Adapt them to your own policy.

Allow search-style AI crawlers but opt out of model training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Here OAI-SearchBot, Claude-SearchBot and PerplexityBot have no group of their own, so they follow the User-agent: * group.

Block a training crawler from part of the site only

User-agent: GPTBot
Disallow: /members/
Disallow: /research/

Remember that the GPTBot group replaces the * group for that crawler, so repeat any general rules (such as /admin/) inside it if you still want them applied.

Allow everything

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

An empty Disallow means nothing is blocked.

Trade-offs of blocking AI crawlers

There is no universally right answer. The decision depends on your business model and how you value visibility against control.

Reasons some sites block training crawlers

• They do not want their content used to train models without a licence or compensation.
• Their content is their product (for example, paid research or original journalism).
• They want to reduce server load from heavy crawling.

Reasons some sites allow them

• They want their brand, products or expertise to be known to AI assistants.
• Their content is marketing material they want widely distributed.
• They see little practical harm and prefer not to maintain a long block list.

Reasons to think carefully before blocking search crawlers

• Blocking a search-focused crawler such as OAI-SearchBot or PerplexityBot can remove your pages from that product's search answers, according to the operators' descriptions. If AI search sends you visitors or leads, blocking may cost you that traffic.
• Allowing a crawler does not guarantee that you will be cited or ranked in any AI answer. It only removes one barrier.

Limits of any choice

• Content that has already been collected is not removed by changing robots.txt today.
• User-initiated fetchers may not follow robots.txt, according to some operators.
• Crawlers that do not identify themselves, or ignore the protocol, are not affected. Server-level controls (firewall or bot management rules) are needed for those.

How to test your robots.txt

1. Open the file in a private browser window at https://yourdomain/robots.txt to confirm it is publicly accessible and returns the content you expect. Google's documentation suggests this step.
2. Check the Search Console robots.txt report, which Google provides for verified properties, to see whether Google fetched the file and any errors it found.
3. Test specific URLs against specific user agents. Google also publishes an open-source robots.txt parser library for local testing. Our AI crawler access checker shows whether common AI user agents are allowed or blocked for a given URL based on your file.
4. Watch your server logs for the user agents you care about, and verify their identity where the operator publishes IP ranges.
5. Retest after every change. A misplaced Disallow: / under User-agent: * can block your whole site from search engines.

Checklist

• The file is at the root of each host, named robots.txt, UTF-8, and returns a 200 status.
• No sensitive URLs are listed; private content is protected by authentication, not robots.txt.
• Pages you want out of search results use noindex (and are not blocked from crawling, or the noindex cannot be seen).
• You have decided a policy for training crawlers, search crawlers and user-initiated fetchers separately.
• Each crawler-specific group repeats any general rules you still want applied.
• Your sitemap URL is listed.
• You have tested key URLs for Googlebot and your chosen AI user agents.
• You review operator documentation periodically for new or renamed crawlers.

Summary

• Robots.txt asks crawlers not to request certain URLs. It is not access control and does not by itself keep pages out of search results.
• Rules are grouped by user agent; a crawler follows the most specific matching group and ignores the rest.
• Operators document separate user agents for AI training, AI search and user-initiated fetching. At the time of writing these include Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and CCBot.
• Blocking training crawlers and blocking search crawlers have different consequences; decide each deliberately.
• Test after every change and recheck operator documentation regularly.

Create a clean file with the robots.txt generator.

Sources (checked at the time of writing): Google Search Central, "Introduction to robots.txt", "Create and submit a robots.txt file", "How Google interprets the robots.txt specification" and the overview of Google crawlers (developers.google.com); OpenAI crawler documentation (developers.openai.com/api/docs/bots); Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (support.claude.com); Perplexity crawler documentation (docs.perplexity.ai/guides/bots); Common Crawl, CCBot (commoncrawl.org/ccbot).