What Is robots.txt? (With Examples)

A robots.txt file lives at the root of your site (yoursite.com/robots.txt) and tells search-engine crawlers which parts of your site they may or may not request. It guides crawling — but it's not a security tool.

Create one in seconds

Use the robots.txt Generatorto build a valid file with allow/disallow rules and your sitemap, then paste it into your site's root.

The basic syntax

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

• User-agent — which crawler the rules apply to. * means all of them.
• Disallow — a path crawlers should not request. An empty Disallow: allows everything.
• Sitemap — the full URL of your sitemap, so crawlers can find all your pages.

Common examples

• Allow everything: User-agent: * then Disallow: (empty).
• Block the whole site: User-agent: * then Disallow: /.
• Block one folder: Disallow: /private/.

The mistake everyone makes

Disallow does not hide a page from Google. It only asks crawlers not to fetch it — a blocked URL can still appear in results if other sites link to it. To keep a page out of search, use a noindexmeta tag or HTTP header instead (and don't also block it in robots.txt, or Google can't see the noindex). Also never rely on robots.txt to hide sensitive pages — the file is public, so it can actually reveal the paths you're trying to hide.

FAQ

Where does robots.txt go?

In your domain root, reachable at https://yoursite.com/robots.txt. It won't work in a subfolder.

Do I need a robots.txt file?

It's optional, but recommended — at minimum to point crawlers to your sitemap and keep them out of admin or duplicate pages.

Should I add my sitemap?

Yes — the Sitemap line helps search engines discover all your pages, which helps indexing.