Free robots.txt tester
Enter a site or page URL and see whether Googlebot, Bingbot and the main AI crawlers may fetch it, according to the site's robots.txt.
What this checks
The tester reads the file the way Google does, following RFC 9309.
- It fetches the robots.txt at the root of the host, following up to five redirects. A 404 means everything is allowed; a 5xx makes Google treat the site as blocked.
- It picks the group that matches each crawler by name and falls back to User-agent: *. The longest matching path wins, and Allow wins a tie. Wildcards (*) and end anchors ($) are supported.
- It tests your path against Googlebot, Bingbot, OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended and every other crawler.
- It flags common mistakes: blocking the whole site, blocking CSS and JavaScript, no Sitemap line, a relative Sitemap URL, Noindex lines, typos and Crawl-delay, which Google ignores.
Questions
Does robots.txt remove a page from Google?
No. It controls crawling, not indexing. A blocked URL can still show up in results if other pages link to it. To keep a page out of the index, use a noindex meta tag or an X-Robots-Tag header, and let crawlers fetch the page so they can read it.
Which rule wins when Allow and Disallow both match?
The most specific one, meaning the rule with the longest matching path. If both are the same length, Google uses the less restrictive rule, so Allow wins.
Does Google follow Crawl-delay?
No. Googlebot ignores it. Bing and some other crawlers honor it, so it can still be useful if they slow your server down.
Want a sitemap that matches your robots.txt?
GenMap crawls your site, leaves out pages that robots.txt or noindex keep out of search, and writes a valid sitemap.xml.
Generate a valid sitemap free