Robots.txt Generator
Build a valid robots.txt with per-crawler rules — including AI bots like GPTBot and ClaudeBot.
- robots.txt controls crawling, not indexing — use a
noindexmeta tag to keep a page out of results. - It must be served from the domain root, at
/robots.txt. - Blocking a page here also stops crawlers seeing its
noindextag, so do not use both on the same URL. - Crawl-delay is ignored by Googlebot; set the rate in Search Console instead.
Your Data Never Leaves Your Device
Every tool runs entirely in your browser. Nothing you type is uploaded, stored, or logged on our servers.
This robots.txt generator builds the file that tells crawlers which parts of your site they may fetch. It supports multiple rule groups, so you can allow general crawling while blocking specific agents — GPTBot, ClaudeBot, Google-Extended, and CCBot are included as presets, since AI crawlers are separate user agents that a * rule does not necessarily cover.
Worth being clear about: robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, and blocking it actually prevents the crawler from seeing a noindex tag — so the two should never be applied to the same URL. Add a Sitemap: line so every crawler can find your sitemap without it being submitted separately.
FAQ
No. It controls crawling, not indexing. A blocked URL can still be listed if other sites link to it. To keep a page out of results use a noindex meta tag — and do not also block it in robots.txt, or the crawler will never see that tag.
At the root of the domain, exactly at /robots.txt. A file at /subfolder/robots.txt is ignored, and each subdomain needs its own.
Add a group per crawler — GPTBot, ClaudeBot, Google-Extended, and CCBot are the common ones — with Disallow: /. They are separate user agents, so a rule for * does not necessarily stop them, and compliance is voluntary.
It explicitly allows everything for that user agent. "Disallow:" with no path is the opposite of "Disallow: /", which blocks the whole site — an easy and costly typo.
Yes. A Sitemap: line with the absolute URL of your sitemap lets every crawler that reads robots.txt discover it, without you having to submit it to each search engine separately. You can list more than one.