Generate Your Robots.txt File in Seconds
Robots.txt Generator
Tell search engines exactly where to go. Generate a perfect robots.txt file, block AI crawlers, and add your sitemap in one click.
Configure your rules
Changes update the preview instantly.
Your robots.txt
Upload this file to the root of your domain.
What is a robots.txt file?
A robots.txt file is a small text file at the root of your website (yourdomain.com/robots.txt) that tells crawlers which parts of your site they can visit. Search engines like Google and Bing check it before crawling, and most reputable AI crawlers do too. A well-written robots.txt keeps bots out of admin areas, duplicate filter pages and internal search results, so crawl budget goes to the pages that matter.
How to use this robots.txt generator
- Pick a default – allow all robots (recommended for most sites) or block all (staging sites).
- Add your sitemap – paste the full URL of your XML sitemap so crawlers find every page.
- Block private paths – list folders like
/admin/or/checkout/, one per line. - Set rules per bot – block AI crawlers or specific search engines individually.
- Copy or download – upload
robots.txtto your site's root and test it in Google Search Console.
Robots.txt directives explained
| Directive | What it does | Example |
|---|---|---|
User-agent | Names the crawler the rules apply to. * means all. | User-agent: Googlebot |
Disallow | Blocks crawling of a path. Empty value allows everything. | Disallow: /admin/ |
Allow | Permits a path inside a disallowed folder. | Allow: /admin/public/ |
Sitemap | Full URL of your XML sitemap. | Sitemap: https://site.com/sitemap.xml |
Crawl-delay | Seconds between requests. Ignored by Google. | Crawl-delay: 10 |
* and $ | Wildcard and end-of-URL match. | Disallow: /*.pdf$ |
Best practices
- Never block CSS or JavaScript files — Google needs them to render your pages.
- Robots.txt is public. Don't use it to hide sensitive URLs; protect them with a login instead.
- To keep a page out of search results, use a
noindexmeta tag and leave it crawlable. - Paths are case-sensitive:
/Admin/and/admin/are different. - Each subdomain needs its own robots.txt file.
Frequently asked questions
What is a robots.txt file?
A plain text file in your site's root that tells search engine crawlers and other bots which pages or sections they may or may not crawl.
Where do I upload robots.txt?
In the root directory of your domain, so it loads at https://yourdomain.com/robots.txt. It won't work inside a subfolder.
Does robots.txt stop a page from being indexed?
No. It controls crawling, not indexing. A blocked URL can still show in results if other sites link to it. Use a noindex tag on a crawlable page instead.
Can I block AI crawlers like GPTBot?
Yes. Crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot respect robots.txt. Set them to "Block" in the Individual robots section above.
Does Google support Crawl-delay?
No, Googlebot ignores it. Bing, Yandex and some other crawlers honor it.