451 451 Unavailable For Legal Reasons

Defined in RFC 7725: the server is denying access because of a legal demand. A court order, a takedown notice, a regulatory requirement, a sanctions regime.

Mechanically it is a 4xx. Crawlers treat it much as they treat a 403: the resource is unavailable, retry occasionally, drop the URL from the index if it stays that way. There is no special handling for the legal part, and no search engine treats a 451 more gently than any other refusal.

What the code adds is honesty. The spec asks for an explanation in the response body, including who demanded the block where that can be disclosed. A 403 tells nobody anything; a 451 with a body tells a reader why.

What causes it for crawlers

Geographic blocking is the usual one, and it produces a confusing picture. Crawlers fetch from a small number of locations, so a block on one region is invisible to a crawler outside it and total to a crawler inside it. Your index can show the page as fine while a large group of users cannot reach it, or the reverse.

Copyright takedowns applied at the URL level. Regulatory blocks, including the data protection blocks that led some sites to refuse European traffic entirely. Sanctions compliance at the CDN edge, which is often applied by a provider rather than by you, and which people discover by finding a 451 they did not configure.

How to diagnose it

Find out where the block originates. A 451 from your application is one you can explain. A 451 from your CDN is a provider policy, and the reason will be in their dashboard rather than in your logs.

Then check whether it is regional:

curl -sI https://example.com/page
# then the same request from a host in the affected region

If the code depends on the requester's location, note which crawler egress locations fall inside the block. Most operators publish their ranges, and those ranges tell you which side of the line the crawler is on. Googlebot and Applebot both publish the lists you would need.

How to fix it, or not

Often you should not. If a legal demand requires the content to be unavailable, returning 451 is compliance working as intended, and the goal is to scope it correctly rather than to remove it.

Scope it to the URLs actually covered. A whole-site 451 for a demand covering three pages removes far more than required and is hard to reverse cleanly once the index has drained.

Do not use a redirect instead. Sending a blocked request to a generic notice page returns 200 for content that is not there, which is a soft 404, and it hides the fact of the block from everyone including you.

Keep /robots.txt reachable. A 451 on the rules file is the same failure as a 403 on it: crawlers lose your instructions entirely, and the block you wanted becomes broader than you specified. See robots.txt for how a failing rules file is interpreted.