Glossary
One term per page, each with the definition first and the practical consequence second.
-
Canonical URL
The canonical URL is the address you nominate as the real one for duplicate content. How the link rel canonical hint works and why it is a hint, not a rule.
-
Cloaking
Cloaking is serving crawlers different content than users. Where the line sits, why the intent test is unreliable, and the setups that cross it by accident.
-
Crawl budget
Crawl budget is how many URLs a crawler will fetch from your host in a period. What sets it, what wastes it, and why most sites do not have a budget problem.
-
Hydration
Hydration attaches JavaScript behavior to server-rendered HTML in the browser. Why crawlers do not care about it and users do, plus the hydration mismatch.
-
JavaScript rendering
Whether a crawler executes your JavaScript before reading the page. Which crawlers do, what the raw versus rendered comparison shows, and what it costs.
-
llms.txt
llms.txt is a proposed markdown file describing a site for language models. What it contains, what it is not, and which operators have committed to reading it.
-
noindex
The noindex directive tells a search engine to keep a page out of its index. Where to put it, how it differs from Disallow, and why the two cancel out.
-
Prerendering
Prerendering produces finished HTML before the request arrives or at the edge. How it differs from server rendering, and when it becomes cloaking.
-
robots.txt
What robots.txt is, how crawlers match its groups, what a missing or failing file means, and the two mistakes that make a correct-looking file do nothing.
-
Soft 404
A soft 404 is a missing page served with a 200 status. Why crawlers keep requesting it, how search engines detect it, and the correct status code to return.
-
Time to first byte
TTFB is the delay between a request and the first byte of the response. What it includes, why crawlers care about it, and why it is not a ranking factor.
-
X-Robots-Tag
The HTTP header form of the robots meta tag. What it controls, why non-HTML files need it, and the scoping mistake that deindexes more than intended.