Robots.txt Generator
Create a valid robots.txt file with crawl rules and sitemap reference.
Use * for all crawlers, or a name like Googlebot.
How to use the Robots.txt Generator
Step 1 — Pick a starting point
- Use a preset, or set the user-agent yourself.
Step 2 — Add rules
- List paths to disallow or allow, one per line.
Step 3 — Add your sitemap
- Paste your sitemap URL so crawlers can find it.
Step 4 — Deploy
- Download and upload robots.txt to your site root.
Frequently asked questions
It tells search engine crawlers which parts of your site they may or may not request. It lives at your domain root, e.g. example.com/robots.txt
No — that is a common mistake. Disallow stops crawling, but a blocked page can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta tag instead.
Allow all opens the whole site, Block all closes it (useful for staging), and WordPress defaults block admin and search paths while allowing the admin-ajax endpoint.
Yes. Adding a Sitemap line helps crawlers find your sitemap even if you have not submitted it in Search Console. Use the full absolute URL.
Save it as robots.txt in your site root so it is reachable at example.com/robots.txt. It must be at the root — a subfolder will not work.
About the Robots.txt Generator
This tool creates a valid robots.txt file — the small text file at the root of a site that gives search engine crawlers instructions about which areas they should and should not request. You set a few rules and it produces correctly formatted output you can download and deploy.
What robots.txt is for
When a well-behaved crawler visits your site, the first thing it checks is robots.txt. The file lets you keep crawlers out of areas that have no business in search — admin screens, internal search results, cart and checkout pages, duplicate parameter URLs — which conserves your crawl budget and keeps low-value pages from being requested repeatedly. It is organised into groups: each begins with a User-agent line naming the crawler the rules apply to (an asterisk means all of them), followed by Disallow and Allow lines listing paths.
An important limitation to understand
Disallow controls crawling, not indexing, and conflating the two is the most common robots.txt mistake. If you Disallow a page, crawlers will not fetch its content — but if other pages link to it, Google can still list the URL in results, just without a description. So robots.txt is the wrong tool for hiding a page from search. To genuinely keep a page out of the index, allow it to be crawled and add a noindex meta tag, or protect it behind authentication. Use robots.txt for crawl management, not for secrecy.
Presets and deployment
The presets give you sensible starting points: Allow all leaves the site fully open, Block all closes everything (handy for a staging site you do not want indexed), and WordPress defaults block the admin and internal search paths while keeping the admin-ajax endpoint reachable, which some plugins need. Add a Sitemap line with your full sitemap URL so crawlers can discover it directly. Save the result as robots.txt in your site root — it only works at the root, not in a subfolder. Everything is generated in your browser.