Meta-WebIndexer

What does Meta-WebIndexer do?

Meta-WebIndexer crawls and indexes public web pages for Meta AI search. It feeds the indexing pipeline that allows Meta AI to surface and cite content in search responses. When Meta AI references your content, it includes citation links that can drive referral traffic back to your site.

Should I allow and optimize for Meta-WebIndexer to drive organic growth?

Yes, allow and optimize for Meta-WebIndexer. Meta AI search cites and links to source pages in its responses, creating a direct referral traffic channel. Meta AI is integrated across Facebook, Instagram, WhatsApp, and Messenger, giving it a massive user base. Blocking this crawler removes your content from Meta AI search results entirely, eliminating a growing source of AI-driven traffic.

Data Usage & Training

Meta describes Meta-WebIndexer as an indexing crawler for Meta AI search, not explicitly for model training. A separate crawler token, Meta-ExternalAgent, is associated with training-related activities. However, Meta's documentation does not explicitly rule out that indexed content could be repurposed for training, so the boundary is not fully clear.

How Meta-WebIndexer Accesses Content

Here's how Meta-WebIndexer accesses your site and understands your content:

• Fetches HTML via standard HTTP requests
• Identifies as meta-webindexer/1.1 in the user-agent string
• Respects robots.txt Allow and Disallow directives
• Processes sitemaps when provided
• Originates from Meta's AS32934 IP ranges

Meta-WebIndexer performs ongoing, regular indexing for Meta AI search. Meta does not publish a precise schedule. Independent sources classify it as relatively low-volume compared with major web search crawlers.

How to Block or Control Meta-WebIndexer

To block Meta-WebIndexer via robots.txt: User-agent: meta-webindexer Disallow: / Robots.txt changes may take up to 24 hours to take effect due to caching delays. For IP-based blocking, verify source IPs against Meta's AS32934 using whois lookups or Meta's geofeed at https://www.facebook.com/peering/geofeed. There is no single static IP list published specifically for Meta-WebIndexer. For issues, contact [email protected]. Note that other Meta fetchers (like Meta-ExternalFetcher and FacebookExternalHit) may bypass robots.txt for user-initiated fetches or security checks, so blocking meta-webindexer does not block all Meta bot traffic.

Common Issues & Troubleshooting

Watch out for these common problems when working with Meta-WebIndexer:

• User-agent strings are easy to spoof; verify requests by checking source IPs against Meta's AS32934 rather than relying on the UA alone
• Meta uses large, changing IP ranges announced via AS32934 rather than a small static list, making IP allowlisting operationally complex
• Robots.txt changes can take up to 24 hours to take effect due to caching
• Other Meta fetchers (Meta-ExternalFetcher, FacebookExternalHit) may still access your site for user-initiated fetches or security checks even if you block meta-webindexer
• No documented support for Crawl-delay, so you cannot throttle crawl rate through robots.txt alone