robots.txt explained: rules, testing, and the classic mistakes

robots.txt is a plain text file with outsized consequences. It's the first thing a crawler fetches from your site, its syntax looks trivial, and a one-character mistake in it can deindex your business. It's also persistently misunderstood — most notably about what it does not do.

What does robots.txt actually do?

It tells crawlers which URLs they may fetch. That's the entire function. Each group of rules starts with one or more User-agent lines naming who it applies to, followed by Disallow and Allow patterns:

User-agent: *
Disallow: /admin/
Allow: /admin/public-docs/

Sitemap: https://www.example.com/sitemap.xml

Two details people miss:

• Consecutive User-agent lines share one rule set. User-agent: * followed by User-agent: Googlebot followed by Disallow: /x blocks /x for both. Many hand-rolled parsers (and humans) get this wrong.
• The longest matching pattern wins, and Allow wins ties. Disallow: /search with Allow: /search/about blocks /search but permits /search/about, because the Allow pattern is longer. Test any path and user-agent combination with our robots.txt tester, which applies the same longest-match rule Google does.

Does robots.txt remove pages from Google?

No — this is the big one. Blocking a URL prevents crawling, not indexing. If other sites link to a blocked page, Google can index the bare URL anyway, showing it with no description ("No information is available for this page").

Worse: if you add a noindex tag to a page and block it in robots.txt, the noindex can never be seen — you've locked the door and posted the eviction notice inside. To remove a page from search results: allow it to be crawled, serve noindex, and wait for the recrawl.

What are the classic robots.txt mistakes?

The staging block that shipped. Disallow: / — correct on staging, catastrophic in production. It's the first thing to check when a site's traffic falls off a cliff after a launch.

Blocking CSS and JS. Google renders pages; blocking /assets/ or /wp-includes/ means it renders yours broken, and judges mobile-friendliness on the wreckage.

Case and pattern confusion. Paths are case-sensitive (/Admin/ ≠ /admin/), Disallow: /page blocks /page, /pages, and /page-two (it's a prefix), and $ anchors the end (Disallow: /*.pdf$ blocks PDFs only).

No Sitemap: line. Free discovery help, most sites omit it. Validate the sitemap you declare with the sitemap checker.

Should I block AI crawlers like GPTBot?

You can — GPTBot (OpenAI training), ClaudeBot, PerplexityBot, Google-Extended (Gemini training) all honor their User-agent groups. Whether you should is strategy, not hygiene.

The trade: blocking keeps your content out of training sets and AI answers; allowing means assistants can cite and recommend you, which is a growing discovery channel. For most businesses trying to be found, being quotable is worth more than being withheld — a blocked site can't be recommended. Publishers monetizing content directly may reasonably decide otherwise. Either way, decide deliberately: check what your file currently says with the robots.txt tester, because plenty of sites are blocking (or allowing) AI crawlers by accident via User-agent: *.