Meta-ExternalAgent

Operator

Meta

User agent

meta-externalagent/1.1

Robots token

meta-externalagent

Obeys robots.txt

Yes, per the operator

Verification

No published method. Meta publishes no IP ranges, no ASN list and no reverse DNS method for its crawlers. Its documentation mentions allowlisting by user agent or IP address without providing the addresses, so no honest verification is possible.

Operator docs

https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/

Meta documents two forms of the same string. The short one:

meta-externalagent/1.1

The longer form appends a parenthesized link to Meta's crawler documentation. Both carry the same identifier, so match the meta-externalagent substring and ignore the tail.

The robots.txt token is meta-externalagent, written lowercase in Meta's own example. Token matching is case-insensitive, so the casing is cosmetic.

Three agents, three behaviors

meta-externalagent crawls for what Meta describes as use cases such as training foundation AI models and improving products by indexing content directly.

meta-externalfetcher is user-triggered, fetching a page because someone asked for it.

facebookexternalhit generates link previews. This is the one that renders the card when your URL is pasted into a Meta app.

Meta notes that the fetcher and facebookexternalhit may skip robots.txt rules, on the same user-initiated reasoning Perplexity gives for Perplexity-User. It does not list that exception for meta-externalagent.

The verification gap

This is the part worth reading twice. Meta publishes no IP ranges, no ASN, and no reverse DNS hostname for any of these agents. Its documentation suggests allowlisting by user agent string or by IP address, then never says which addresses.

So there is no honest way to confirm that a request claiming to be meta-externalagent came from Meta. The string is a claim, and copying a string costs nothing. Every other large operator on this site publishes at least one check: Googlebot and CCBot publish both DNS and ranges, GPTBot, ClaudeBot and Amazonbot publish ranges. Meta publishes neither.

The practical consequence is narrow but real. If you allow meta-externalagent by user agent at your edge, you have allowed anyone who types it. If you block it by user agent, you block the honest version and miss anything that renames itself. Either rule is a preference, not a control.

Does it honor robots.txt

Meta documents robots.txt as the mechanism for blocking these crawlers and does not claim an exception for meta-externalagent. Because there is no verification method, you also cannot audit compliance: a log line showing the string is not proof that Meta sent it, so an apparent violation might be someone else entirely.

Allow it

User-agent: meta-externalagent
Allow: /

Block it

User-agent: meta-externalagent
Disallow: /

Keep link previews working while blocking the crawler:

User-agent: meta-externalagent
Disallow: /

User-agent: facebookexternalhit
Allow: /

That second group matters. A rule written against meta or facebook as a broad prefix catches the preview fetcher too, and the symptom is a Facebook or Instagram post showing your link as a bare URL with no title, no description and no image. People usually connect that back to the robots.txt edit days later.

What blocking costs you

Training-corpus inclusion, and whatever Meta means by indexing content directly. Neither produces attributed traffic, so there is no number to watch.

The cost that does show up is the one above: losing link previews by writing the rule too broadly. If social sharing matters to you, test it after the change by pasting a URL into a Meta app and looking at the card. Check the response your server actually returns too, since a 403 served by bot mitigation to any non-browser agent will kill previews without a single line in your robots.txt.

Find it in your logs

awk '$0 ~ /meta-external|facebookexternalhit/ {print $1, $12}' access.log | sort | uniq -c | sort -rn | head

Print the agent field, because the three Meta tokens behave differently and the address tells you nothing. With no published ranges there is no next step after this command. You can count the traffic and you can decide what to do with it, and you cannot attribute it.

What to do with an unverifiable crawler

Treat the decision as being about the string rather than about the operator. Blocking meta-externalagent at the edge stops every request carrying that string, which includes Meta's and includes anyone borrowing it. That is a coherent outcome even though it is not the one the rule appears to describe.

What you should not do is build reporting on it. A dashboard counting Meta crawl volume is counting a self-reported label, and any conclusion about Meta's behavior drawn from it is unsound. Where verification exists, as with Googlebot or GPTBot, the same dashboard is worth building.

Check your site against Meta-ExternalAgent