403 403 Forbidden

The server understood the request and refuses to fulfil it. Unlike a 401, no authentication will help, and unlike a 404 the resource is not being described as missing. The answer is no.

For a crawler that is a clear signal, which is exactly why an accidental 403 is so expensive.

What causes it for crawlers

Bot mitigation, in the large majority of cases. A WAF, CDN rule or security product refuses requests that lack browser-shaped headers, do not run a JavaScript challenge, or arrive without a cookie set by a previous challenge. No crawler passes any of those, so a rule aimed at scrapers takes Googlebot with it.

Rate limiting configured to refuse rather than defer is the next most common cause, and it is the wrong code for the job. A 429 says come back later. A 403 says never.

After that: geographic blocking that catches crawler egress, allowlists that were never updated when an operator changed its ranges, and directory permissions that deny everything without an index file.

How to diagnose it

Ask as a crawler, not as yourself:

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/120.0.0.0 Safari/537.36" https://example.com/

A 403 here with a 200 in your browser is the whole diagnosis. Repeat it for /robots.txt, which is the worst version of this bug: a 403 on the rules file means your rules are not being read at all, so every careful robots.txt group you wrote is doing nothing.

Then check whether it is agent-based or address-based. Send the same crawler user agent from a different network. If the 403 follows the string, it is a user agent rule. If it follows the address, it is a range or reputation rule, and your CDN's log is where the reason lives.

Search Console reports these as crawl anomalies rather than as 403s specifically, which is why they often run for weeks before anyone notices.

How to fix it

Allowlist verified crawlers rather than user agent strings. Verification exists for most operators worth allowing: forward-confirmed reverse DNS for Googlebot, Bingbot and CCBot, published IP ranges for GPTBot, ClaudeBot and Amazonbot. An allowlist keyed on the string alone admits every impersonator, which is the reason the rule was written in the first place.

Exempt /robots.txt from every challenge, unconditionally. There is nothing in it worth protecting and everything to lose by hiding it.

If rate is the problem, return a 429 with a Retry-After header instead. Compliant crawlers back off and come back, which is the outcome you wanted.

When a 403 is correct

For traffic you have decided to refuse, and for a legal block a 451 says the same thing with a reason attached. Bytespider publishes no verification method and does not reliably honor robots.txt, so an edge rule returning 403 is the only thing that works. That is the code doing its job.