Free robots.txt & Sitemap Checker
Fetch and analyze a site’s robots.txt and sitemap.xml — count the Disallow rules, flag sensitive paths (admin, backup, .git, config) that leak structure to attackers, detect a site-wide block, and confirm the sitemap and its declaration.
By scanning, you confirm you own or have permission to test this domain. Not a substitute for manual penetration testing.
What this robots.txt checker checks
How it works
Read-only· fetches robots.txt + sitemap.xml- 1We fetch /robots.txt and /sitemap.xml with plain GETs.
- 2We count Disallow rules, flag sensitive paths they leak (admin, backup, .git, config), detect a whole-site block, and summarise the sitemap and its child sitemaps.
- 3Only those two standard files are requested.
Why it matters
robots.txt is public, so listing admin, backup or config paths in it hands attackers a map of where to look — robots rules are not a security control. A healthy sitemap, meanwhile, helps search engines index your real pages.
Frequently asked questions
Is the robots.txt checker free?
Yes — no sign-up. We fetch /robots.txt and /sitemap.xml and summarize what they reveal.
Is it safe to list private paths in robots.txt?
No. Because robots.txt is publicly readable, listing sensitive directories advertises them. Protect those paths with authentication and access controls instead of relying on Disallow.
Does robots.txt improve security?
Not on its own — it only requests that compliant crawlers skip paths. Treat it as an SEO/crawler tool, never as an access control.