Free tool · no signup
Robots.txt Tester
Fetch and parse any site's robots.txt, test whether a specific path is blocked for a specific crawler, and catch rules that hide your pages from Google. Free, no signup.
What robots.txt does — and doesn't do
robots.txt tells crawlers which paths they may fetch. It does not remove pages from the index — a blocked URL can still be indexed from links pointing at it, showing a bare "no information available" result. To keep a page out of results, use a noindex directive and make sure the page isn't blocked, so crawlers can actually see it. This tester fetches any site's robots.txt, parses the groups correctly (including multi-agent groups, which many parsers get wrong), and gives a verdict for any path and user-agent using the same longest-match rule Google applies.
Common mistakes it catches
A stray Disallow: / that blocks the whole site; rules in the wrong group so Googlebot isn't covered; a missing Sitemap: line; and blanket blocks on AI crawlers you may actually want citing you. Verify the sitemap it declares with the sitemap checker, then run a full page check to see what crawlers find once they're allowed in.