Blocked by robots.txt: what it means
In your audit this appears as URLs blocked by robots.txt
Your site's robots.txt file forbids crawlers from downloading this page. It is not that they do not want it: you are not letting them in to see it.
Why it matters
If a search engine cannot read the page, it cannot know what it is about, see its content, or follow the links leading out of it. Everything inside is out of reach.
There is a nuance that surprises many people: blocking in robots.txt does not guarantee the page stays out of Google. If other sites link to it, it can still be listed — but with no description and a note that it could not be read. The worst of both worlds.
When it is the right thing
Blocking is fine for whatever adds nothing to the index and burns crawl budget: admin panels, internal search results, URLs with endless filters, carts. The question is whether this page belongs in that category.
How to fix it
- Open
yourdomain.com/robots.txtand find theDisallowrule that catches this address. - If the page should be crawled, remove that rule or narrow it so it does not include it.
- Check you do not have a stray
Disallow: /: that line blocks the entire site, and it is a classic accident when moving from staging to production.
Use noindex on the page, and do not block it in robots.txt. Google needs to get in to read that instruction. Blocked, it never sees it.
Does your site have this problem?
Analyse your page for free and we will tell you which of these issues you have, and on which pages.
Audit a site