Fix Guide
How to Fix a Missing Robots.txt
/robots.txt is missing or unreachable.
Recommended fix
Publish a simple robots.txt file that allows crawlable sections and points to the sitemap.
This guide is designed to pair with a scanner report. Run a URL scan first, then follow the implementation steps.
Developer task
Create /robots.txt with User-agent: * and Sitemap: https://example.com/sitemap.xml.
Implementation steps
How to apply this fix
- Create a plain-text robots.txt at the site root.
- Add User-agent: * and only disallow paths that must stay out of search.
- Add a Sitemap: https://example.com/sitemap.xml line pointing to the live sitemap.
- Publish it at /robots.txt and confirm it returns HTTP 200 as text/plain.
Verify
Re-run the URL scan after deploying the change to confirm the issue clears and the Themerella Score updates.
Related guides