How to Write a robots.txt File

robots.txt is a plain text file that tells search engine crawlers which parts of your site they may visit. It is the first thing Googlebot requests. A correct one takes five lines.

Create the file at your site root

The file must live at the root of your domain and be named exactly robots.txt, all lowercase:

https://yourdomain.com/robots.txt

Crawlers only look there. A robots.txt in a subfolder does nothing.

Allow crawling

The minimum file that lets every crawler visit every page:

User-agent: *
Allow: /

User-agent: * means the rules apply to all crawlers. Allow: / means the whole site is open. If you have nothing to block, this is a complete file.

Block what should not be indexed

Add Disallow lines for paths crawlers should skip. Common ones are admin panels, search result pages, and internal API routes:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /search

Paths are prefixes, so /admin/ blocks everything under it.

One warning: robots.txt is a request, not a lock. Well-behaved crawlers honor it, but it does not hide anything from people. Never rely on it for private content. Use authentication for that.

Point to your sitemap

Add one line with the full, absolute URL of your sitemap:

Sitemap: https://yourdomain.com/sitemap.xml

This is the single most useful line in the file. It tells crawlers exactly which pages exist so they do not have to discover them by following links.

Put it together

A complete, production-ready robots.txt:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Test it

Open https://yourdomain.com/robots.txt in a browser. You should see the plain text, not a 404 or an HTML page.

Then in Google Search Console, open Settings, then the robots.txt report. It shows whether Google fetched the file successfully and flags any lines it could not parse.