robots.txt Generator
Create a robots.txt file for your website.
Generate a correct robots.txt for your website in seconds. Choose whether to allow all crawlers or block the whole site, list the paths you want to keep out of search, add an optional crawl-delay, and include your sitemap URL. Copy the result into a robots.txt file at your site's root.
* means all crawlers.
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
Save this as robots.txt in your site's root.
Runs entirely in your browser — nothing is uploaded.
How to use the robots.txt generator
- 1Choose your crawler policy and add any paths to disallow.
- 2Add your sitemap URL (recommended).
- 3Copy the generated robots.txt into your site's root.
Frequently asked questions
What is a robots.txt file?
It's a text file at your site's root (yoursite.com/robots.txt) that tells search-engine crawlers which pages or folders they may or may not request. It guides crawling — it is not a security or privacy control.
Where do I put robots.txt?
In the root of your domain, so it's reachable at https://yoursite.com/robots.txt. It won't work in a subfolder.
Does Disallow hide a page from Google?
Not reliably — Disallow asks crawlers not to fetch a page, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta tag or header instead.
Should I add my sitemap?
Yes — adding a Sitemap: line helps search engines discover all your pages, which is good for indexing.