robots.txt is a plain-text file at your domain root that tells search-engine crawlers which paths they may or may not request. It also points crawlers to your XML sitemap. It controls crawling, not indexing.
Located at /robots.txt, this file gives crawl instructions to bots. Used well it protects private or duplicate areas and advertises your sitemap; used carelessly it can accidentally hide your entire site from search.
Check your website
See how your site handles robots.txt: guiding crawlers without blocking your site — free, no account needed.
For business owners
A single wrong line in robots.txt can remove your whole website from Google. It is one of the highest-risk small files on your site. Done right, it helps search engines spend their crawl budget on your important pages and find your sitemap. Every professional site should have a deliberate, tested robots.txt.
How it works (technical)
The file uses simple directives grouped by user-agent:
User-agent: *\nDisallow: /admin/\nAllow: /\nSitemap: https://example.com/sitemap.xml
Key points: Disallow blocks crawling of a path but does not guarantee a page stays out of the index — use noindex meta tags for that. A stray Disallow: / blocks the entire site. Always include the absolute URL of your sitemap.
Real-world example
A team pushed a staging robots.txt containing Disallow: / to production during a deploy. Within days Google dropped most pages from the index and traffic collapsed. Correcting the file and requesting re-crawling restored rankings, but the outage cost weeks of traffic.
Why it matters
robots.txt directly controls whether search engines can crawl your site. Scanners check that it exists, is valid, does not accidentally block everything, and references your sitemap.
How to fix it
Create
/robots.txtat your domain root if it is missing.Allow crawling of public content; only Disallow genuinely private or duplicate paths.
Add a
Sitemap:line with the absolute URL of your XML sitemap.Never ship a
Disallow: /from staging to production.Test with a robots.txt tester and re-scan.
Best practices
Keep it minimal and intentional — every Disallow should have a reason.
Use noindex meta tags (not robots.txt) to keep pages out of search results.
Reference your sitemap so crawlers discover new pages faster.
Common mistakes
Shipping
Disallow: /from a staging environment.Assuming Disallow removes a page from Google (it only blocks crawling).
Blocking CSS/JS that search engines need to render the page.
Frequently asked questions
Does Disallow hide a page from Google?
No. It stops crawling, but a blocked URL can still appear in results if linked elsewhere. Use a noindex meta tag to remove it from the index.
Where must robots.txt live?
At the root of your domain: https://example.com/robots.txt. It cannot be in a subfolder.
Put this into practice
See how your site scores with Plexa Trust — start with a free scan, then unlock the complete audit on Pro.
Scan your website free