New
AI Crawler Checker
See which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) your robots.txt allows or blocks.
How it works
Robots.txt Generator
Build a valid robots.txt without memorizing the syntax. Start from a preset, add rules for all bots or specific ones, list your sitemaps, decide how AI crawlers are treated, and test any URL before you upload the file.
Rules
- 01
Pick a preset such as WordPress or Online shop, or start from Allow everything.
- 02
Add Disallow paths for areas crawlers shouldn't fetch, with Allow exceptions, and a group for any bot that needs different rules.
- 03
Choose how AI crawlers are treated and list your sitemap URLs.
- 04
Test a few URLs, then copy or download the file and upload it to the root of your site.
About this tool
What robots.txt does, and what it doesn't
robots.txt is a plain-text file at the root of your site (example.com/robots.txt) that tells crawlers which paths they may fetch. Rules are grouped by user-agent: a group for * applies to every bot that doesn't have its own group, and a bot with its own group follows only that group. Within a group, the most specific (longest) matching rule wins, and Allow wins a tie, as described in RFC 9309 and Google's documentation. That's how this generator's tester decides, too.
robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it; use a noindex meta tag or header to keep a page out of the index, and don't block that page in robots.txt or crawlers won't see the noindex. It's also a request, not access control: well-behaved crawlers follow it, but anything private needs a password.
Groups per bot
Rules for every crawler, plus separate groups for Googlebot, Bingbot or any user-agent you name.
AI crawler controls
Block AI training crawlers while staying in AI search, or block all AI bots, using an up-to-date token list.
Built-in tester
Check whether a path is allowed for Googlebot, GPTBot and others, and see the exact rule that decided it.
Mistake checks
Warns about full URLs instead of paths, relative sitemaps, duplicate groups and accidentally blocking the whole site.
FAQ
Frequently asked questions
Where do I put robots.txt?
At the root of each host: https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain (such as shop.example.com) needs its own.
Does Disallow remove a page from Google?
No. It stops crawling, not indexing. To remove a page from search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header.
Should I block CSS and JavaScript?
No. Google renders pages like a browser, and blocking the files it needs can hurt how your pages are understood and ranked.
Does Google support Crawl-delay?
No. Google ignores it and adjusts its crawl rate automatically. Bing and Yandex honor it.
Missing something?
Tell us the tool you wish existed.
Describe the job and the file types involved. The most requested tools get built first.