What Is robots.txt? Definition and Example
A text file that tells search engine crawlers which pages they may crawl.
What is robots.txt?
The robots.txt file is a text file in the root directory of a website that gives instructions to search engine crawlers. It controls which areas may be crawled and which may not.
Example: a shop does not want crawlers to spend time on its checkout pages. Its robots.txt contains the lines User-agent: * and Disallow: /checkout/, plus a Sitemap: line pointing to its XML sitemap.
Key directives
- User-agent – defines which crawler the rule applies to
- Disallow – blocks specific paths from being crawled
- Allow – permits crawling of a path that was blocked before
- Sitemap – points to the XML sitemap
Important: robots.txt does not prevent indexing. Pages can still be indexed despite a Disallow rule if external links point to them. Use noindex for real protection against indexing.
A wrong rule in robots.txt can keep search engines away from important pages, so the file is a standard part of every technical SEO review. Our article on llms.txt and structured data covers related files aimed at AI systems.
Frequently asked questions
What is robots.txt?
robots.txt is a text file in the root directory of a website. It tells search engine crawlers which areas they may crawl and which they may not.
Why does robots.txt matter for SEO?
It controls how crawlers move through your site, and a wrong rule can block important pages. Note that robots.txt does not prevent indexing; use noindex for that.
What is an example of robots.txt?
A shop adds the lines "User-agent: *" and "Disallow: /checkout/" so that crawlers skip its checkout pages, plus a Sitemap line pointing to its XML sitemap.