Robots.txt Generator Online
Build a robots.txt file for your website in seconds. Set user-agent rules, specify disallowed and allowed paths, and add your sitemap URL. No signup, runs entirely in your browser.
⏱ 6 min read · Complete guide below
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
How the Robots.txt Generator Works
- 1Enter your sitemap URL (optional but recommended for SEO).
- 2Click Add Rule and choose a user-agent — use
*for all bots, or a specific name likeGooglebot. - 3Enter the paths you want to disallow or allow. Use the quick-path buttons to insert common values like
/admin/or/api/. - 4Click Copy to copy the generated robots.txt, then paste it into your site's root directory as
robots.txt.
robots.txt Syntax Reference
The Robots Exclusion Protocol supports a small set of directives. Each user-agent block starts with User-agent: followed by the bot name, then one or moreDisallow: or Allow: lines. An empty Disallow:line means allow everything for that user-agent. Paths are case-sensitive and must start with /.
What robots.txt Does — and What It Doesn't
The single most important thing to understand about robots.txt is what it actually controls. It is a crawling instruction, not an indexing one, and not a security measure. Disallowing a path asks well-behaved crawlers not to fetch those URLs, which saves crawl budget and keeps bots away from areas like admin panels or internal search results. But it does not reliably keep a page out of search results — if other sites link to a disallowed URL, it can still appear in the index (often with no description, because the crawler was not allowed to read it). To actually remove a page from search, use a noindex meta tag or the X-Robots-Tag header on the page itself, and be aware that a page must be crawlable for a crawler to see that noindex instruction.
Understanding User-Agents and Rule Precedence
Each block in a robots.txt file targets a user-agent — the name a crawler identifies itself by. User-agent: * sets default rules for every compliant bot, while naming a specific agent such as Googlebot or GPTBot lets you give it its own rules. A crawler follows the block that most specifically matches its name, and ignores the others, so a bot with its own named block will not also apply the * rules. Within a block, the Allow directive can carve out exceptions to a broader Disallow — for instance disallowing an entire folder but allowing one public file inside it. Getting this precedence right is where most robots.txt mistakes happen.
Controlling AI Crawlers
A newer and increasingly common use of robots.txt is opting out of AI training data collection. Several AI companies publish named crawlers — such as GPTBot, ClaudeBot, Google-Extended, and CCBot — that you can disallow independently of ordinary search crawlers. This lets you keep your site fully visible in Google and Bing while declining to have its content used to train language models, or vice versa. As with all robots.txt rules, this relies on the crawler choosing to honour the file, which the major, reputable AI crawlers currently do. Add a separate block for each AI user-agent you wish to restrict.
Robots.txt Best Practices
Always Include a Sitemap
The Sitemap: directive tells search engines where to find your XML sitemap, helping them discover all your pages faster.
Block AI Crawlers Selectively
Add separate rules for GPTBot, ClaudeBot, and CCBot to opt out of AI training data collection without affecting Googlebot.
Disallow Admin Paths
Block /admin/, /login, and /dashboard to save crawl budget and keep internal pages out of search indices.
Use Wildcards Carefully
The * wildcard matches any sequence of characters. Disallow: /*.json$ blocks all URLs ending in .json. Test with Google Search Console after deploying.
Validate After Deployment
Use Google Search Console's robots.txt tester to verify your rules work as expected before they affect crawling in production.
Don't Rely on It for Security
robots.txt is a courtesy document. Malicious bots ignore it. Protect sensitive content with authentication, not just a Disallow directive.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of your website (e.g. https://example.com/robots.txt) that instructs web crawlers which pages or directories they are allowed or not allowed to access. It follows the Robots Exclusion Protocol.
Does robots.txt prevent pages from appearing in search results?
No. Disallowing a URL in robots.txt prevents crawlers from fetching that page, but it does not prevent the URL from appearing in search results if other pages link to it. To remove a URL from search results entirely, use a noindex meta tag or the X-Robots-Tag HTTP header on the page itself.
What is the difference between Disallow and Allow?
Disallow tells the crawler not to access the specified path. Allow overrides a broader Disallow rule for a specific sub-path. For example, you can Disallow /private/ while Allow-ing /private/public-file.pdf within the same user-agent block.
What does "User-agent: *" mean?
"User-agent: *" applies the rules to all web crawlers that honour the Robots Exclusion Protocol. You can add specific rules for individual bots — such as Googlebot, Bingbot, or GPTBot — by using their specific user-agent name.
Will blocking a bot with robots.txt stop it from crawling?
Well-behaved crawlers (Googlebot, Bingbot, etc.) respect robots.txt voluntarily. Malicious bots or scrapers may ignore it entirely. For security-sensitive content, use authentication or server-level access controls, not robots.txt.
Where should I place my robots.txt file?
The file must be placed at the root of your domain — https://www.example.com/robots.txt. Subdirectory placement (e.g. /blog/robots.txt) is not supported by the protocol and will be ignored.
Is my data private when using this tool?
Yes. Everything runs in your browser. No data is sent to any server. The tool generates the robots.txt content locally from your inputs.
What is the difference between robots.txt and a noindex tag?
They control different things. robots.txt controls crawling — whether a bot is allowed to fetch a URL — while a noindex meta tag or X-Robots-Tag header controls indexing — whether a page may appear in search results. Confusingly, if you block a page in robots.txt, crawlers cannot read its noindex tag, so the URL can still be indexed via external links. To keep a page out of search, allow it to be crawled and add noindex, rather than disallowing it.
How do I block AI crawlers like GPTBot or ClaudeBot?
Add a separate user-agent block for each AI crawler you want to restrict and disallow the paths (or the whole site with Disallow: /). Common AI crawler names include GPTBot, ClaudeBot, Google-Extended, and CCBot. Because these blocks are independent of the ones for Googlebot and Bingbot, you can opt out of AI training data collection while keeping your site fully visible in normal search results. The major AI crawlers currently honour these rules.
Can robots.txt secure or hide private content?
No. robots.txt is a public file that anyone can read, and it only asks compliant crawlers to stay away — malicious bots and scrapers ignore it entirely. Listing a sensitive path in robots.txt can even advertise its existence. For anything that must be private, use proper protection such as authentication, access controls, or server-side restrictions. Treat robots.txt as a crawl-management courtesy, never as a security mechanism.
Do I need a robots.txt file at all?
Not strictly. If you are happy for crawlers to access your entire site, you can omit the file and crawlers will simply crawl everything they can find. A robots.txt becomes useful when you want to steer crawl budget away from low-value areas (like admin pages, faceted-search URLs, or internal scripts), point crawlers to your sitemap, or manage AI crawlers. Even a minimal file with just a sitemap reference is a reasonable, low-risk addition for most sites.