The SEO crawler that won't stay quiet about your site's problems.
Checks robots.txt, the robots meta tags and the X-Robots-Tag headers against the crawlers that matter, on any page. No account, nothing stored.
Whether a crawler can read a page is decided in three places at once, and any one of them is enough to keep it out:
robots.txt, which says what paths each agent may request.meta robots tag inside the document.X-Robots-Tag header on the response, the one almost nobody checks because it does not show in the page source.This tool reads all three and tests them against a list of named crawlers: search engines, AI agents, SEO tools and social networks. It is two requests, however long the list.
They are different things and they get confused constantly. Disallow says "do not request this URL"; noindex says "you may read it, but do not show it". A page blocked in robots.txt can still appear in Google, without a description, if somebody links to it: Google knows it exists and was never allowed in to read the noindex you would have put there.
My page is blocked by robots.txt and still shows in Google. That is the case above. If you want it gone, allow the crawl and add noindex — not the other way round.
Does blocking GPTBot hurt my rankings? No. GPTBot and ClaudeBot are training crawlers; search uses Googlebot, and they are separate lists.
Can robots.txt hide something? No. It is a public file listing exactly what you would rather nobody looked at. Private things need a password.