Meta-ExternalFetcher

What does Meta-ExternalFetcher do?

Meta-ExternalFetcher performs on-demand fetches of web pages when a user triggers a request through Meta products, including Meta AI and other agentic features. It retrieves content so Meta's AI can navigate or display web information during user interactions. Because fetches are user-initiated, there is potential for in-product linkouts or surfaced source links, though the exact citation behavior is not documented.

Should I allow and optimize for Meta-ExternalFetcher to drive organic growth?

Allowing Meta-ExternalFetcher lets Meta AI and agentic features retrieve and display your content to users who are actively looking for it. This can drive referral traffic through in-product linkouts or surfaced source links, though Meta has not documented exactly how or when citations appear. Given Meta's massive user base across Facebook, Instagram, WhatsApp, and Meta AI, even modest citation behavior can translate into meaningful visibility. Blocking this bot may prevent your content from appearing in Meta AI responses entirely.

Data Usage & Training

It is unclear whether content fetched by Meta-ExternalFetcher is retained or used for model training. Meta's documentation describes the bot as supporting agentic and on-demand fetches but does not explicitly address training data usage. If training data collection is a concern, Meta's separate crawlers (like CCBot or FacebookBot) are more likely candidates, but you should contact [email protected] for clarification.

How Meta-ExternalFetcher Accesses Content

Here's how Meta-ExternalFetcher accesses your site and understands your content:

• Fetches HTML via standard HTTP requests triggered by user actions
• Identifies as meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
• May bypass robots.txt Disallow rules for user-initiated fetches
• Caches robots.txt for up to 24 hours before re-checking
• JavaScript rendering capability is unknown

Strictly on-demand. Meta-ExternalFetcher only visits your site when a user action in a Meta product triggers a fetch. There is no continuous or scheduled crawling.

How to Block or Control Meta-ExternalFetcher

Add the following to your robots.txt to block Meta-ExternalFetcher: User-agent: meta-externalfetcher Disallow: / However, Meta's documentation states that this bot may bypass Disallow rules for user-initiated fetches. Robots.txt changes can also take up to 24 hours to take effect due to caching. For stronger control, use IP-based blocking. Verify request IPs against Meta's AS32934 address space using their geofeed at https://www.facebook.com/peering/geofeed. Be aware that Meta's IP ranges are large and change over time. For sensitive content, the most reliable approach is requiring authentication, session tokens, or CAPTCHAs. You can also contact [email protected] to discuss crawler behavior directly.

Common Issues & Troubleshooting

Watch out for these common problems when working with Meta-ExternalFetcher:

• Robots.txt Disallow rules may not prevent user-initiated fetches, as Meta-ExternalFetcher can bypass them
• User-Agent blocking is unreliable because UA strings can be spoofed
• IP-based blocking is difficult to maintain because Meta's IP ranges are large and change frequently
• Robots.txt changes take up to 24 hours to take effect due to caching
• JavaScript-heavy pages may not render correctly (rendering capability is undocumented)
• Content behind login walls or CAPTCHAs will not be fetched