How to use the AI Crawler Access Checker
- Enter the URL of the page you want to check.
- We fetch robots.txt from the site root and apply the rules for each AI crawler token to that URL path.
- Review Allowed or Blocked for each crawler, with the exact rule and line that decided it.
- Check page-level meta robots and X-Robots-Tag values, then use the robots.txt generator to change your rules.
Worked example
Checking https://example.com
example.com returns 404 for /robots.txt, so under RFC 9309 every crawler is allowed: all 19 AI tokens show Allowed with the reason "No robots.txt (4xx), so access is allowed."
How it works
robots.txt is fetched from the URL’s origin (redirects followed, 500 KB limit) and parsed with our RFC 9309 implementation: case-insensitive product token matching, the most specific group (combined when repeated), longest-match precedence with allow winning ties, * and $ wildcards and percent-encoding normalisation. A 4xx response means everything is allowed; a 5xx or network error means compliant crawlers treat the site as disallowed. The crawler list, purposes and documentation links were verified against each operator’s official documentation on 2026-09-25.
Assumptions
- Only tokens documented by their operators are included.
- Results show what compliant crawlers should do, not what every bot actually does.
Frequently asked questions
Does blocking AI crawlers stop my content being used?
robots.txt is a voluntary standard. Operators that follow it will not crawl blocked paths, but it cannot technically prevent access, and content copied elsewhere may still be collected.
Do user-triggered fetchers follow robots.txt?
It depends on the operator. For example, OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores robots.txt, while Anthropic says its bots honour it. Each crawler row links to the operator’s documentation.
Will blocking Google-Extended affect Google Search?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether content is used for Gemini models.
Does allowing AI crawlers mean AI tools will cite my page?
No. Allowing access only makes crawling possible. No one outside those companies controls whether a page is used or cited in AI answers.
Limitations
- Crawler tokens change; the list reflects operator documentation on 2026-09-25.
- Does not detect firewall or CDN bot blocking, which can block crawlers regardless of robots.txt.
- Limited to 30 checks per visitor every 10 minutes.