Sitemap vs robots.txt: what is the difference?
By Pablo Murad. Last updated
A sitemap lists the URLs you want search engines to find. A robots.txt file tells crawlers which URLs they may not fetch. The first helps discovery, the second limits crawling, and neither one removes a page from search results.
Sitemap vs robots.txt at a glance
| Sitemap | robots.txt | |
|---|---|---|
| Purpose | Tells search engines which URLs exist on your site, so they can discover them. | Tells crawlers which URLs they can access. Its main use is to avoid overloading your site with requests. |
| Format | XML, RSS 2.0 or mRSS, Atom 1.0, or a text file with one URL per line. | Plain text in UTF-8, with the fields user-agent, allow, disallow and sitemap. |
| File name and location | Any name. Google recommends the site root. It affects only URLs under its own directory unless you submit it in Search Console. | Must be named robots.txt and sit in the top-level directory. It applies only to the host, protocol and port that serve it. |
| Size limit | 50 MB uncompressed or 50,000 URLs per file. | 500 KiB. Content after that is ignored. |
| Effect on crawling | A hint. Google does not guarantee it will download the sitemap or crawl every URL in it. | Controls what compliant crawlers fetch. Following it is voluntary: it is up to the crawler to obey. |
| Effect on indexing | None by itself. A listed URL is not guaranteed to be indexed. | None that you can rely on. A blocked URL can still be indexed if other sites link to it, usually without a description. |
| How search engines find it | Search Console, the Search Console API, or a Sitemap line in robots.txt. | Crawlers request it from the root of the host. |
| Keeps a page out of Google | No. | No. Google says to use noindex or password protection. |
| Required | No. Small, well-linked sites can do without one. | No. If the file returns a 4xx error (except 429), Google treats it as if no robots.txt existed. |
Sources: Google, introduction to robots.txt, Google, robots.txt specification, Google, build and submit a sitemap and the sitemaps.org protocol. Checked October 21, 2026.
What a sitemap does
A sitemap is a list. It points search engines at pages they might not reach through links, and it can add details about video, images and news. Google calls it a way to crawl a site more efficiently, and it also says the file doesn't guarantee that all the items in it will be crawled and indexed. The XML sitemap guide covers the format.
What robots.txt does
A robots.txt file is a set of access rules. A Disallow line asks crawlers to stay out of a path, and an Allow line makes an exception. Google's own description is that the file tells crawlers which URLs they can access so your site is not overloaded with requests. Compliance is voluntary, and some crawlers ignore the file.
robots.txt is not a noindex tag
The most common mix-up is to block a page in robots.txt and expect it to disappear from Google. It will not. Google states that robots.txt is not a mechanism for keeping a web page out of Google: a disallowed URL can still be indexed when other sites link to it, and it may appear without a description. To keep a page out of search results, use noindex or password protection. Google also warns that crawling and indexing rules can counteract each other when you combine them.
How a sitemap and robots.txt work together
- Point crawlers to the sitemap from robots.txt. A line such as
Sitemap: https://www.example.com/sitemap.xmltakes the full URL, is independent of any user agent, may point to another host, and you can add as many as you need. - Do not list URLs you have blocked. Search Console reports "Sitemap contains urls which are blocked by robots.txt" when a rule contradicts the file.
- Do not block the sitemap itself. Google respects robots.txt when fetching sitemaps, so a rule that matches the file makes the report show "Couldn't fetch".
User-agent: *
Disallow: /cart/
Sitemap: https://www.example.com/sitemap.xmlHere the cart is off limits to crawlers and the sitemap is announced to every search engine that reads the file. The free robots.txt tester shows whether a given URL is allowed or blocked by your rules.
Which one should you use?
- You want a page found: link to it internally and list it in the sitemap.
- You want crawlers to spend less time on a section: disallow it in robots.txt, and leave it out of the sitemap.
- You want a page out of search results: add noindex or password protection, and let crawlers fetch the page so they can see the rule.
The two files are independent: a sitemap works without a robots.txt file, and robots.txt works without a sitemap. If you publish both, make sure they do not contradict each other. To check what your sitemap contains, use the free sitemap validator.
Generate a sitemap that lists only the pages you want indexed.
Crawl a site