Robots.txt Generator
Build a robots.txt file with crawler rules, sitemaps and AI bot controls, then test any URL against it.
See your site the way your sitemap describes it. Enter a domain or paste the XML to get a browsable folder tree, the biggest sections, how recently pages changed, and a list of problems worth fixing.
Enter a site or paste a sitemap
You'll see every URL as a folder tree, which sections are biggest, how fresh the pages are and what to fix.
How it works
Enter a domain such as example.com, or a sitemap's full URL. You can also paste sitemap XML.
Read the summary: URL count, largest sections and how fresh the pages are.
Open folders in the tree to explore, or filter to find specific URLs.
Fix the problems listed, and download the CSV if you need the full list.
About this tool
An XML sitemap lists the URLs a site wants search engines to know about, optionally with a lastmod date for each. Looking at it as a tree shows the structure behind those URLs: which sections hold most of the pages, how deep the folders go, and whether the pages you care about are listed at all. It's also a quick way to spot thin or forgotten sections before an audit or a migration.
This visualizer finds the sitemap from robots.txt or the usual locations, follows sitemap indexes (up to 10 child sitemaps and 5,000 URLs), and checks for common problems: duplicates, http:// URLs, query strings and fragments, capital letters, mixed trailing slashes, invalid or future lastmod dates and URLs from other hosts. Pasted XML is analyzed entirely in your browser.
Every URL grouped by path, with page counts and a bar showing each folder's share of the site.
The largest first-level sections, and pages by lastmod year, at a glance.
Duplicates, http URLs, parameters, case, trailing slashes, bad dates and stray hosts, with examples.
Search every URL and download the list as CSV for a spreadsheet.
FAQ
Usually at /sitemap.xml or /sitemap_index.xml, and often listed in robots.txt as a Sitemap: line. Enter just your domain and the visualizer looks in those places.
To keep fetching fast and fair, live sitemaps are read up to 5,000 URLs and 10 child sitemaps. For larger sites, paste individual sitemap files.
Google says it uses lastmod when it's consistently accurate, to decide when to recrawl. Set it to when the page content last changed, not to the time the sitemap was generated.
The address is sent to our fetching service, which downloads only that site's public robots.txt and sitemap files and returns the URLs to your browser. Nothing is stored. Pasted XML never leaves your browser.
Missing something?
Describe the job and the file types involved. The most requested tools get built first.