AI Crawler Checker
See which of 22 AI crawlers, from GPTBot to PerplexityBot, your robots.txt allows or blocks.
Find out whether AI search engines and assistants can read, understand and cite a page. Enter a URL or paste HTML for a scored checklist with specific fixes.
Enter a URL or paste HTML
You'll get a score and a checklist of what helps or blocks AI search and assistants.
Checking
How it works
Enter a page URL, or paste the page's HTML if fetching isn't available.
Review the score and the checklist. Failing and warning items explain what to change.
Fix the issues, then check again to compare.
About this tool
AI search tools and assistants such as ChatGPT search, Perplexity, Claude and Google's AI features can only cite what they can fetch and understand. Most of their crawlers don't run JavaScript, so content that only appears after scripts run can be invisible to them. Clear headings, semantic HTML and structured data make it easier to pick out facts and answer questions, and robots.txt decides whether they may visit at all.
This checker looks at the page as an AI crawler would: the text present in the raw HTML, heading structure, landmarks, title and description, canonical and robots meta tags, Open Graph and JSON-LD. With a URL, it also checks the site's robots.txt for 22 AI crawlers, looks for llms.txt and a sitemap, and shows the page converted to clean Markdown, close to what a language model reads. Each check explains why it matters and what to change.
Scores the HTML before JavaScript runs, which is what most AI crawlers get.
Content, structure, metadata and AI access, each with a specific fix.
AI crawler access in robots.txt, llms.txt and sitemap, from a URL.
See the page as clean Markdown, close to what a model reads.
FAQ
Generative engine optimization (GEO) is making content easy for AI search and assistants to find, understand and cite. It builds on regular SEO: crawlable pages, clear structure and accurate metadata matter most.
If the main content is rendered in the browser, crawlers that don't run JavaScript see an almost empty page. Server-side rendering or static generation fixes this.
No. It means there are no technical barriers. Whether a page is cited depends on its content, authority and the question being asked.
Yes: open the page, view its source, and paste the HTML. Site-level checks need a public URL.
The page itself plus the site's robots.txt, sitemap and llms.txt, fetched by our service and returned to your browser, where the scoring runs. Nothing is stored, and private network addresses are refused.
Missing something?
Describe the job and the file types involved. The most requested tools get built first.