X-Robots-Tag

Also called x robots tag, robots header

An HTTP response header carrying the same directives as the robots meta tag:

X-Robots-Tag: noindex, nofollow

It applies to any response, which is the whole reason it exists. A PDF has no head element, so a meta tag is not an option and the header is. The same goes for images, generated files and API responses that you would rather not see in an index.

You can target a single crawler by prefixing the token:

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: bingbot: noindex, nofollow

Directives include noindex, nofollow, none, noarchive, nosnippet and unavailable_after, and they behave the same as their meta tag equivalents. See noindex for the one that matters most.

The mistake

Setting it at the wrong level of the server config and not noticing.

The header is usually added in nginx, Apache or a CDN rule, and those rules are matched by path or content type rather than by page. A rule intended for /downloads/*.pdf written one directory too high covers the whole section. A rule matching Content-Type: application/pdf catches the PDF viewer page too if that page is served with the wrong type. Neither mistake produces an error, a warning, or anything visible in the HTML.

Check the live header rather than the config:

curl -sI https://example.com/report.pdf | grep -i x-robots-tag

The second trap is the familiar one. A URL blocked in robots.txt is never fetched, so its X-Robots-Tag is never read, exactly as with a meta tag. The header only works on responses a crawler is allowed to request, which means Googlebot has to be able to reach the file for the header to do anything at all.