Quick answer: robots.txt is a text file at the root of your domain that tells crawlers which URLs they may request. It controls crawling, not indexing: a URL blocked in robots.txt can still appear in Google if other pages link to it. To keep a page out of search results, allow crawling and use a noindex tag instead. Google does not support noindex or crawl-delay rules inside robots.txt.
robots.txt is a small file with a big blast radius. One wrong line can remove a whole site from Google. I have seen staging rules copied to production, a “Disallow: /” left after a redesign, and CSS and JavaScript folders blocked so Google could not render pages properly. This guide covers what the file does, what it does not do, and the rules I use.
What robots.txt does
Before crawling a site, well-behaved crawlers such as Googlebot request /robots.txt. The file lists rules by user agent. A simple example:
User-agent: *
Disallow: /cart/
Disallow: /search/
Allow: /
Sitemap: https://www.example.com/sitemap_index.xmlThis tells all crawlers not to crawl the cart and internal search pages, allows everything else, and points to the sitemap.
Crawling vs indexing: the most common misunderstanding
| Goal | Right tool | Wrong tool |
|---|---|---|
| Keep a page out of search results | noindex meta tag or X-Robots-Tag header | robots.txt Disallow |
| Stop crawlers wasting time on endless URLs | robots.txt Disallow | noindex alone |
| Consolidate duplicates | Canonical tag or redirect | robots.txt Disallow |
| Protect private content | Password protection | robots.txt Disallow |
The key point: Google cannot see a noindex tag on a page it is not allowed to crawl. If you block a page in robots.txt and add noindex, the noindex is never read. To remove a page from the index, allow crawling until Google has processed the noindex.
Syntax rules that trip people up
- Location: the file must be at the root, such as https://www.example.com/robots.txt. Each subdomain and protocol needs its own file.
- Paths are case-sensitive.
/Admin/and/admin/are different. - Wildcards: Google supports
*for any sequence of characters and$for the end of a URL.Disallow: /*.pdf$blocks PDF files. - Most specific rule wins. For Google, when Allow and Disallow conflict, the longer, more specific rule applies.
- Size: Google reads up to 500 KiB of the file and ignores anything after that.
- Unsupported rules: Google ignores
noindex,nofollowandcrawl-delayin robots.txt.
What to block and what not to block
Usually worth blocking:
- Internal site search results.
- Cart, checkout and account pages.
- Faceted filter combinations that create near-infinite URLs. See my guide to faceted navigation SEO before doing this, because it depends on which filters have search demand.
- Tracking or session parameters that create duplicate URLs.
Do not block:
- CSS, JavaScript and image files needed to render your pages.
- Pages you want removed from the index (use noindex instead).
- Pages carrying canonical tags or redirects you need Google to see.
robots.txt and AI crawlers
Many site owners now use robots.txt to manage AI crawlers. Common user agents include GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and CCBot (Common Crawl). Google offers a separate token, Google-Extended, which controls whether content is used for Gemini models and grounding. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
Blocking Googlebot itself would remove you from Search and from AI Overviews, which use the same index. Decide on AI crawlers deliberately: blocking them can protect content, but it also reduces the chance of being cited in AI assistants. My llms.txt guide covers the related question of AI-specific files.
How to test robots.txt
- Open the robots.txt report in Search Console to see the version Google fetched, when, and any errors.
- Use the URL Inspection tool on important URLs to confirm they are not blocked.
- Before deploying changes, test the new rules against a list of key URLs with a robots.txt testing library or crawler.
- After a site launch or migration, check robots.txt on the live domain first. Staging rules are a classic launch-day disaster.
Server errors matter too. If robots.txt returns a 5xx error, Google may slow or pause crawling of the site until it can read the file again.
robots.txt is one of the first things I check in a technical SEO audit. If you are unsure whether your rules are helping or hurting, my technical SEO service includes a full crawl and indexing review.
Frequently asked questions
Can robots.txt remove a page from Google?
No. It only stops crawling. Use a noindex tag, or the Removals tool in Search Console for urgent temporary removal.
Does every site need a robots.txt file?
No. Without one, crawlers assume everything is allowed. A simple file with a sitemap line is still good practice.
How quickly does Google see robots.txt changes?
Google generally caches robots.txt for up to 24 hours, so changes usually take effect within a day.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews are part of Google Search and use Googlebot. Google-Extended controls use of content for Gemini models and grounding outside Search.
Sources: Google Search Central: Introduction to robots.txt; How Google interprets the robots.txt specification; Overview of Google crawlers and fetchers.
