Robots.txt checker

What the robots.txt checker does

This tool requests the robots.txt file of a given website, reports whether the file exists, and shows the sitemap references and disallow rules it contains. It gives you a quick, readable summary of the crawl instructions a site publishes to search engine bots, without having to open the raw file and parse it yourself.

Robots.txt is a plain text file placed at the root of a domain, for example https://example.com/robots.txt. Crawlers that respect the Robots Exclusion Protocol read it before fetching other pages. The file can allow or disallow specific paths for specific user agents and can point to XML sitemaps. Because it sits at a fixed location, a missing or malformed file can quietly affect how a site is crawled, which is why checking it is a routine part of technical SEO audits.

How to read the results

The checker reports the exact URL it requested, so you can confirm that the file was looked up at the domain root rather than at a subpath. If the file is found, the tool extracts the sitemap entries and the disallowed paths it detects and lists them separately. This makes it easy to see whether the site advertises its sitemaps and which areas it asks crawlers to skip.

If the file is not found, that does not automatically mean something is broken. A site without a robots.txt file is treated by compliant crawlers as if everything is allowed. Still, an unexpected 404 on a domain that should have the file is worth investigating, because it may indicate a server configuration issue, a redirect chain, or a file that was removed during a deployment.

The disallowed paths list shows the path patterns the file restricts. Read them with the user agent context in mind: a rule that applies to one bot may not apply to another, and rules are matched as prefixes, not as exact URLs. A single Disallow: / entry, for instance, blocks everything for the user agents it targets.

Why robots.txt matters for SEO

Robots.txt is the first line of crawl control. It helps prevent crawlers from wasting budget on admin panels, internal search results, staging copies, faceted navigation, or duplicate parameter URLs. When it is configured well, crawlers spend more time on pages that deserve indexing. When it is configured poorly, important pages can be blocked by accident, and the resulting drop in visibility may take weeks to diagnose.

The file also serves as a discovery aid. Sitemap directives inside robots.txt give crawlers an additional way to find XML sitemaps, alongside search console submissions. This is not a substitute for a submitted sitemap, but it is a useful redundancy.

Common mistakes to avoid

• Blocking CSS and JavaScript. If crawlers cannot fetch the resources a page depends on, they may misjudge the rendered page. Modern search engines render pages, so blocking stylesheets and scripts is rarely a good idea.
• Using robots.txt to hide pages from search results. A disallowed URL can still be indexed if other pages link to it, because crawlers may index the URL without visiting it. Use noindex meta tags or HTTP headers for that purpose instead.
• Wildcards and anchors used incorrectly. Support for * and $ is widespread but not universal across all bots, so test the patterns you rely on.
• Stale rules after a redesign. Old disallow entries often survive migrations and end up blocking new, important sections.
• Case and trailing slash confusion. Paths are case-sensitive, and a missing or extra trailing slash changes what a rule matches.

Practical checks to run

Start by confirming that the file loads with a 200 status and is served as plain text. Then verify that the sitemap URL listed in the file actually resolves and returns valid XML. Review each disallow rule and ask whether the blocked path should really be excluded from crawling. Pay attention to rules aimed at specific bots such as image or news crawlers, since a broad rule may unintentionally restrict them. Finally, compare the live file with what you expect from your repository or CMS, because production and staging environments frequently drift apart.

Limitations

The checker reports what the file contains at the moment of the request. It does not simulate how every search engine will interpret each rule, and it does not verify whether blocked pages are still indexed. Use it as a fast diagnostic starting point, then confirm crawl behavior in the search engine webmaster tools you rely on.