noindex

Also called meta robots noindex, noindex tag

A directive telling a search engine not to include a page in its index. Two places to put it. As a meta tag in the HTML head:

<meta name="robots" content="noindex">

Or as an X-Robots-Tag HTTP header, which is the only option for a PDF, an image or any response that is not HTML:

X-Robots-Tag: noindex

A noindexed page is still crawled, still followed for links unless you add nofollow, and still consumes crawl budget.

The mistake

Combining it with a Disallow rule, which produces the opposite of the intended effect.

# robots.txt
User-agent: *
Disallow: /private/

<!-- /private/page -->
<meta name="robots" content="noindex">

The crawler obeys the first rule, never fetches the page, and never sees the second. The URL can then stay indexed, listed from whatever links point at it, with no snippet because no content was read. The page you most wanted hidden keeps the most durable listing.

Pick one. To remove a page: allow the crawl, serve noindex, wait for the recrawl, and only then add the Disallow if you also want to save the crawl requests. To block a crawl and accept a bare URL listing: use robots.txt alone.

The other frequent failure is scope. A noindex applied through a template or a CDN rule can cover far more than intended, and nothing in your analytics distinguishes "traffic fell" from "half the site left the index". Check the header on a live URL, not the configuration that is supposed to set it.