Free tool
robots.txt checker
Paste a domain or a robots.txt URL. See Allow and Disallow rules, sitemap links, and the mistakes that change what crawlers fetch.
- Free
- No signup
- Nothing stored
How it works
- 01
Enter a domain or a robots.txt URL
A bare domain is fetched at /robots.txt. A URL that already ends in robots.txt is fetched as written.
- 02
Read the groups
Each User-agent lists Allow and Disallow paths. Sitemap lines are listed separately.
- 03
Fix the flagged rows
Start with fails: a file that is HTML, a User-agent * that disallows the whole site, and a Sitemap that is not an absolute URL.
A missing file means allow all
robots.txt is a request to polite crawlers. A 404 means there is no request, so crawlers may fetch any public URL. An empty 200 does the same. The useful file names a User-agent, lists the paths that agent should skip, and points at the sitemap with an absolute URL.
Disallow is not a lock
Disallow: /private asks a crawler not to fetch that path. It does not hide the URL. People can still open it, and a crawler that ignores the file can still request it. Do not put secrets behind a Disallow line. User-agent: * with Disallow: / asks every unnamed crawler to skip the whole site, which is the mistake this checker treats as a fail.
Sitemap lines have to be absolute
Sitemap: https://example.com/sitemap.xml is valid. Sitemap: /sitemap.xml is not. The longer matching path wins when Allow and Disallow overlap, and an equal Allow beats a Disallow. For which AI crawlers you want allowed, run the AI SEO checker. The publishing notes are in Next.js SEO for AI search.
