AI Readiness Checker
Score how well a page can be read, understood and cited by AI search and assistants.
Find out which AI bots can read your site. Check a live robots.txt or paste your own, test any page path, and generate rules to block AI training while staying visible in AI search.
Check a site or paste a robots.txt
You'll see whether each AI crawler can reach the path, and which rule decided it.
| Crawler | Purpose | Access | Decided by |
|---|
How it works
Enter your domain to fetch its robots.txt, or paste the file.
Optionally enter a page path to test. Each AI crawler shows allowed or blocked, with the rule that decided it.
Use the generator to create rules for your policy and add them to your robots.txt.
About this tool
AI companies use different crawlers for different jobs. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train models. AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so assistants can cite and link to them. User-triggered fetchers such as ChatGPT-User and Claude-User load a page because someone asked an assistant about it. Blocking them all removes your site from AI answers, not just from training data.
This checker reads robots.txt the way the standard (RFC 9309) and Google describe: it finds the group for each crawler's token, falls back to the * group, and applies the most specific matching rule, with Allow winning ties. Some tokens, such as Google-Extended and Applebot-Extended, aren't separate crawlers but control how content already crawled for search is used for AI. robots.txt is a request, not an enforcement mechanism; well-behaved crawlers follow it.
From OpenAI, Anthropic, Google, Apple, Perplexity, Meta, Amazon, Common Crawl, Mistral and seven others, grouped into training, search and user fetches.
RFC 9309 groups, wildcards, $ anchors and longest-match precedence.
See the exact rule and line that allows or blocks each crawler.
Block AI training only, block all AI, or allow all, ready to paste.
FAQ
Add “User-agent: GPTBot” followed by “Disallow: /” to robots.txt. To stay in ChatGPT search results, don't block OAI-SearchBot.
No. Google-Extended only controls use in Gemini models and grounding. Googlebot and Search indexing aren't affected.
No. robots.txt only affects future crawling. Content already collected may remain in existing training data.
Major AI companies say their crawlers do, with some exceptions for user-initiated fetches. Less reputable scrapers may ignore it; for those you need server or firewall rules.
Only that site's public robots.txt, downloaded by our fetching service and handed back to your browser. Nothing is stored. You can paste a robots.txt instead if the site isn't public.
Missing something?
Describe the job and the file types involved. The most requested tools get built first.