robots.txt & sitemap.xml Generator
Build a valid robots.txt with per-crawler rules, or generate a sitemap from a list of URLs.
- robots.txt controls crawling, not indexing — use a
noindexmeta tag to keep a page out of results. - It must be served from the domain root, at
/robots.txt. - Blocking a page here also stops crawlers seeing its
noindextag, so do not use both on the same URL. - Crawl-delay is ignored by Googlebot; set the rate in Search Console instead.
Your Data Never Leaves Your Device
Every tool runs entirely in your browser. Nothing you type is uploaded, stored, or logged on our servers.
This generator builds both files that tell crawlers how to treat your site. The robots.txt side supports multiple rule groups, so you can allow general crawling while blocking specific agents — GPTBot, ClaudeBot, Google-Extended, and CCBot are included as presets, since AI crawlers are separate user agents that a * rule does not necessarily cover.
Worth being clear about: robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, and blocking it actually prevents the crawler from seeing a noindex tag — so the two should never be applied to the same URL. On the sitemap side, paste a list of URLs and the XML is generated with proper entity escaping. Keep lastmod honest: it is the field search engines actually act on, while priority and changefreq are largely ignored.
FAQ
No. It controls crawling, not indexing. A blocked URL can still be listed if other sites link to it. To keep a page out of results use a noindex meta tag — and do not also block it in robots.txt, or the crawler will never see that tag.
At the root of the domain, exactly at /robots.txt. A file at /subfolder/robots.txt is ignored, and each subdomain needs its own.
Add a group per crawler — GPTBot, ClaudeBot, Google-Extended, and CCBot are the common ones — with Disallow: /. They are separate user agents, so a rule for * does not necessarily stop them, and compliance is voluntary.
It explicitly allows everything for that user agent. "Disallow:" with no path is the opposite of "Disallow: /", which blocks the whole site — an easy and costly typo.
Barely. Google has said it largely ignores both. An accurate lastmod is the field that genuinely influences recrawling, so keep it truthful rather than always setting it to today.
50,000 URLs or 50 MB uncompressed, whichever comes first. Beyond that, split into multiple sitemaps and reference them from a sitemap index file.