To fix a missing robots.txt, create a plain-text file at https://yoursite.com/robots.txt with User-agent: *, Allow: / for public pages, any needed Disallow rules for private paths, and a Sitemap: line pointing to your XML sitemap. Upload to your site root and re-scan.
robots.txt is the first file many crawlers request. Without it, search engines lack crawl guidance and may not discover your sitemap efficiently. Creating a deliberate, minimal robots.txt takes minutes and is a baseline technical SEO requirement.
Check your website
See how your site handles how to fix a missing robots.txt file — free, no account needed.
Business impact
A missing robots.txt will not usually block indexing entirely, but it signals an unfinished site to auditors and misses an easy chance to point Google at your sitemap and protect private areas. For any business that depends on search traffic, it is basic hygiene.
Why this happens
robots.txt is missing when nothing is served at /robots.txt (404) or the file was never created after launch. Common on new sites, staging clones, and single-page apps where only the app is deployed. The file must live at the domain root — not in a subfolder — and be plain text, not HTML.
How to confirm the issue
Manually: visit https://yoursite.com/robots.txt in your browser. A 404 means it is missing. Google Search Console → Settings → robots.txt also reports fetch errors.
With Plexa Trust: run a scan and look for "robots.txt Missing". Re-scan after publishing the file.
Step-by-step fix
Create a plain-text file named
robots.txt.Add
User-agent: *andAllow: /for public content.Add
Disallow:only for paths you genuinely want blocked (e.g./admin/).Add
Sitemap: https://yoursite.com/sitemap.xmlwith your real sitemap URL.Upload to your web root so it is served at
/robots.txt.Verify with curl or your browser, then re-scan.
Platform-specific fixes
WordPress
Many SEO plugins (Yoast, Rank Math) generate robots.txt automatically — enable it in plugin settings.
Or upload
robots.txtto your site root via FTP/SFTP (same folder as wp-config.php).If using a caching plugin, purge cache after adding the file.
Cloudflare / static hosting
Place
robots.txtin your static site's public root (e.g./publicor/dist).On Netlify/Vercel, put it in the project root; it deploys automatically.
Apache / Nginx
Place the file in your document root (e.g.
/var/www/html/robots.txt).Ensure no rewrite rule intercepts
/robots.txtand returns HTML instead.
Shopify / Wix / Squarespace
Shopify: edit via Online Store → Preferences or theme; check platform docs for robots.txt editing.
Wix/Squarespace: platforms usually provide a default robots.txt — verify it exists and add sitemap reference if missing.
How to verify the fix
Never use
Disallow: /on a production site unless you intend to block all crawling.Include your sitemap URL so crawlers discover pages faster.
Re-scan with Plexa Trust and confirm "robots.txt Missing" is cleared.
Common mistakes
Shipping
Disallow: /from a staging template.Putting robots.txt in a subfolder instead of the domain root.
Using robots.txt to hide pages from the index (use noindex meta tags instead).
Frequently asked questions
Is robots.txt required for Google to index my site?
No, but it is strongly recommended. It guides crawlers and advertises your sitemap without blocking public content.
What is a safe starter robots.txt?
User-agent: * / Allow: / / Sitemap: https://yoursite.com/sitemap.xml — then add Disallow rules only where needed.
Can robots.txt block my whole site from Google?
Yes. Disallow: / blocks crawling of everything. Double-check before deploying.
Does Disallow remove pages from search results?
No. It blocks crawling, not indexing. Use a noindex meta tag to keep pages out of results.
Where exactly must the file live?
At the root: https://example.com/robots.txt — not in /blog/ or any subfolder.
Should I block /wp-admin/?
Disallowing /wp-admin/ is common on WordPress, but Allow /wp-admin/admin-ajax.php if your theme needs it.
How often should I update robots.txt?
When you add private sections, change your sitemap URL, or restructure paths worth blocking.
How do I confirm it worked?
Visit /robots.txt, check Search Console, and re-scan with Plexa Trust.
Think you've fixed it?
Run a free scan to verify the issue is resolved. Upgrade to Pro on Plexa Trust for the full audit, monitoring alerts, and score history.
Verify with a free scan