What robots.txt actually does — and doesn't
robots.txt is a plain-text file at your domain root (yoursite.com/robots.txt) that tells well-behaved crawlers which paths they're allowed to request. It is a request, not a security boundary — it doesn't remove pages from an index, and it doesn't stop crawlers that choose to ignore it. Two consequences follow directly from that:
- Disallowing a URL doesn't guarantee it disappears from search results. If other pages link to that URL, Google can still index the URL itself (without crawling its content) based on those links. Use
noindexmeta tags — not robots.txt — to actually keep a page out of the index. - Never rely on robots.txt to hide sensitive content. Anyone can read your robots.txt file directly, and disallowed paths are often the first thing an attacker checks.
A safe baseline template
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?filter=
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
This blocks crawlers from wasting budget on admin routes, cart/checkout flows, internal search result pages, and faceted-navigation URL parameters (a very common source of near-infinite duplicate URLs on ecommerce and directory sites), while explicitly allowing everything else and pointing crawlers to your sitemap.
The most common robots.txt mistakes
Disallow: /left in from staging. The single most damaging robots.txt mistake: a blanket disallow rule that blocks the entire site from being crawled, often left over from a staging environment and never removed after launch. Check this first on any new site audit.- Blocking CSS and JS files. Search engines render pages like a browser now, and blocking the assets needed to render the page correctly can hurt how it's evaluated. Never disallow
/assets/,/*.css, or/*.jswholesale. - Forgetting the sitemap reference. Not required, but it's a free, zero-risk way to point every well-behaved crawler straight to your full URL list.
- Using robots.txt to fight duplicate content instead of canonical tags. Faceted navigation and parameterized URLs are usually better handled with
rel=canonicalpointing to the clean URL, combined with selective robots.txt rules for the worst offenders — not a robots.txt rule alone. - Case sensitivity errors. Paths in robots.txt are case-sensitive.
/Admin/and/admin/are different rules entirely.
Testing before you deploy
Always validate a new robots.txt file before pushing it live — a single misplaced Disallow: / can silently deindex an entire site over the following weeks as crawlers respect it and gradually drop pages. At minimum:
- Fetch the live file at
/robots.txtafter deploying and confirm it matches what you intended. - Spot-check that your most important pages (homepage, key landing pages, product pages) aren't caught by any disallow rule.
- Confirm the sitemap URL resolves and returns your actual sitemap, not a 404.
How this fits into a full technical SEO setup
Robots.txt is one piece of crawl control — pair it with a clean XML sitemap, correct canonical tags, and a consistent internal linking structure. None of these fix content quality problems; they just make sure the content you do want indexed is discoverable and not wasted on low-value paths.
FAQ
Do I need a robots.txt file at all? Not strictly — if none exists, crawlers assume everything is allowed. But an explicit file is still worth having so you can deliberately manage crawl budget and reference your sitemap.
Can robots.txt block specific bots only?
Yes. Use a specific User-agent line (e.g. User-agent: GPTBot) instead of * to target rules at a particular crawler, such as blocking AI training crawlers while still allowing search engine crawlers.
Will blocking a page in robots.txt remove it from Google's index if it's already there? Not reliably, and often the opposite — Google can't re-crawl the page to see a noindex tag if robots.txt blocks it. To remove an indexed page, allow crawling, add a noindex tag, wait for reprocessing, then block if needed.