Crawl budget

Also called crawl budget optimization, crawl rate

The number of URLs a crawler will fetch from one host over a period. It is the product of two independent things: how fast the crawler thinks it can fetch without hurting your server, and how much it wants your pages at all.

The rate half responds to your server. Fast, consistent responses raise it. Timeouts, 503 responses and slow time to first byte lower it, sometimes for longer than the incident lasted.

The demand half responds to your content. Pages that change, get linked and get clicked are refetched often. Pages nothing points at are visited rarely, however fast your server is.

The mistake

Treating budget as the cause when the real problem is waste.

Almost every site with a crawl budget complaint has a URL space larger than its page count. Faceted navigation multiplies: four filters with five values each is 625 combinations off one category. Session and tracking parameters produce a distinct URL per visitor. Internal search results generate a page per query that anyone ever typed. Calendars generate months forever.

The crawler works through all of it, because from outside there is no way to tell a generated URL from a real one. What looks like a budget shortage is a crawler spending everything you have on URLs you never wanted crawled.

The fixes are ordinary and they are not "ask for more budget":

Block the generated patterns in robots.txt. Return a real 404 where a page does not exist, since a soft 404 keeps the URL in rotation forever. Point duplicates at one canonical URL. Keep redirect chains short, because every hop is a fetch and a redirect loop burns them without end.

Do those and the budget question usually disappears. If it does not, you have a genuinely large site, and then the rate half is worth engineering.