Ahrefs crawler used for backlink and SEO analysis.
- Platform:
- Ahrefs
- User agent:
- ahrefsbot
- Tags:
- SEO
Ahrefs site auditing bot that checks technical SEO issues.
- Platform:
- Ahrefs
- User agent:
- ahrefssiteaudit
- Tags:
- SEO
Allen Institute crawler for AI training datasets.
- Platform:
- AI2
- User agent:
- ai2bot
- Tags:
- AI Training
Enterprise search indexing crawler for Amazon Kendra.
- Platform:
- Amazon
- User agent:
- amazon-kendra
- Tags:
- Search Engine Crawler
Amazon Q assistant fetching content to answer user queries.
- Platform:
- Amazon
- User agent:
- amazon-qbusiness
- Tags:
- AI Training
Amazon crawling agent used for AI training and discovery.
- Platform:
- Amazon
- User agent:
- amazonbot
- Tags:
- AI Training
Crawler for product discovery and search indexing.
- Platform:
- Amazon
- User agent:
- amazonproductdiscoverybot
- Tags:
- E-commerce, Search Engine Crawler
Apify actor that crawls websites and extracts text content for AI models, LLM apps, and RAG pipelines.
- Platform:
- Apify
- User agent:
- apifywebsitecontentcrawler
- Tags:
- AI User Initiated
Google service discovery bot for API-related resources.
- Platform:
- Google
- User agent:
- apis-google
- Tags:
- Search Engine Crawler
Apple web crawler for indexing and AI training.
- Platform:
- Apple
- User agent:
- applebot
- Tags:
- Search Engine Crawler, AI Training
Baidu's primary crawler for web search indexing.
- Platform:
- Baidu
- User agent:
- baiduspider
- Tags:
- Search Engine Crawler
Qwant's SEO crawler for site analysis and indexing.
- Platform:
- Qwant
- User agent:
- barkrowler
- Tags:
- SEO
Microsoft Bing web crawler for search indexing.
- Platform:
- Microsoft
- User agent:
- bingbot
- Tags:
- Search Engine Crawler
ByteDance data collection bot used for AI training.
- Platform:
- ByteDance
- User agent:
- bytespider
- Tags:
- AI Training
Open web crawl used widely in AI training datasets.
- Platform:
- Common Crawl
- User agent:
- ccbot
- Tags:
- AI Training
OpenAI browsing agent fetching pages at user request.
- Platform:
- OpenAI
- User agent:
- chatgpt-user
- Tags:
- AI User Initiated
Microsoft Clarity bot for SEO and verification checks.
- Platform:
- Microsoft
- User agent:
- claritybot
- Tags:
- SEO
Anthropic crawler that indexes content for Claude search.
- Platform:
- Anthropic
- User agent:
- claude-searchbot
- Tags:
- AI Search Index
User-initiated fetches triggered by Claude sessions.
- Platform:
- Anthropic
- User agent:
- claude-user
- Tags:
- AI User Initiated
Claude Code agent fetching pages during development or tooling tasks.
- Platform:
- Anthropic
- User agent:
- claude-code
- Tags:
- Coding, AI User Initiated
Anthropic crawler for collecting training data.
- Platform:
- Anthropic
- User agent:
- claudebot
- Tags:
- AI Training
DataForSEO bot collecting ranking and SERP data.
- Platform:
- DataForSEO
- User agent:
- dataforseobot
- Tags:
- SEO
Moz crawler for link analysis and SEO metrics.
- Platform:
- Moz
- User agent:
- dotbot
- Tags:
- SEO
DuckDuckGo assistant fetching content for answers.
- Platform:
- DuckDuckGo
- User agent:
- duckassistbot
- Tags:
- AI User Initiated
DuckDuckGo crawler for privacy-first web indexing.
- Platform:
- DuckDuckGo
- User agent:
- duckduckbot
- Tags:
- Search Engine Crawler
Exa agent fetching pages at user request for search and research.
- Platform:
- Exa
- User agent:
- exabot
- Tags:
- AI User Initiated
Firecrawl agent fetching pages during user-initiated web scraping tasks.
- Platform:
- Firecrawl
- User agent:
- firecrawlagent
- Tags:
- AI User Initiated
Google agent assisting users with deep research tasks.
- Platform:
- Google
- User agent:
- gemini-deep-research
- Tags:
- AI User Initiated
Google agent navigating the web on behalf of users (e.g. Project Mariner).
- Platform:
- Google
- User agent:
- google-agent
- Tags:
- AI User Initiated
Vertex AI agent used to retrieve and index documents.
- Platform:
- Google
- User agent:
- google-cloudvertexbot
- Tags:
- AI User Initiated
Opt-in crawler for Google AI model training.
- Platform:
- Google
- User agent:
- google-extended
- Tags:
- AI Training
Accessibility agent that reads page content aloud.
- Platform:
- Google
- User agent:
- google-read-aloud
- Tags:
- User Initiated (Non-AI)
Feed fetcher used by Google for RSS/Atom polling.
- Platform:
- Google
- User agent:
- feedfetcher-google
- Tags:
- User Initiated (Non-AI)
Crawler for Google Publisher Center submissions.
- Platform:
- Google
- User agent:
- googleproducer
- Tags:
- User Initiated (Non-AI)
Verification bot for site ownership checks.
- Platform:
- Google
- User agent:
- google-site-verification
- Tags:
- User Initiated (Non-AI)
NotebookLM agent that fetches sources for notes.
- Platform:
- Google
- User agent:
- google-notebooklm
- Tags:
- User Initiated (Non-AI), SEO
Google Store bot for product content and search.
- Platform:
- Google
- User agent:
- storebot-google
- Tags:
- E-commerce, Search Engine Crawler
Primary Google crawler for search indexing.
- Platform:
- Google
- User agent:
- googlebot
- Tags:
- Search Engine Crawler
Google image crawler for indexing visual content.
- Platform:
- Google
- User agent:
- googlebot-image
- Tags:
- Search Engine Crawler
Google video crawler for indexing video pages.
- Platform:
- Google
- User agent:
- googlebot-video
- Tags:
- Search Engine Crawler
Internal Google crawler used for non-public systems.
- Platform:
- Google
- User agent:
- googleother
- Tags:
-
Internal Google image crawler.
- Platform:
- Google
- User agent:
- googleother-image
- Tags:
-
Internal Google video crawler.
- Platform:
- Google
- User agent:
- googleother-video
- Tags:
-
Crawler used by Search Console URL Inspection.
- Platform:
- Google
- User agent:
- google-inspectiontool
- Tags:
- SEO
OpenAI crawler used to gather training data for GPT.
- Platform:
- OpenAI
- User agent:
- gptbot
- Tags:
- AI Training
Naver search crawler used in Korean search index.
- Platform:
- Naver
- User agent:
- yeti
- Tags:
- Search Engine Crawler
OpenAI crawler indexing content for answer engines.
- Platform:
- OpenAI
- User agent:
- oai-searchbot
- Tags:
- AI Search Index
OpenAI crawler used to evaluate ad landing pages for safety and relevance.
- Platform:
- OpenAI
- User agent:
- oai-adsbot
- Tags:
- AI Search Index
Parallel crawler that discovers and indexes websites for search APIs.
- Platform:
- Parallel
- User agent:
- shapbot
- Tags:
- AI Search Index
Similarweb crawler used for competitive web traffic and SEO analysis.
- Platform:
- Similarweb
- User agent:
- similarweb
- Tags:
- SEO
User-initiated fetches triggered from Perplexity sessions.
- Platform:
- Perplexity
- User agent:
- perplexity-user
- Tags:
- AI User Initiated
Perplexity crawler that indexes the public web.
- Platform:
- Perplexity
- User agent:
- perplexitybot
- Tags:
- AI Search Index
Desktop SEO spider used for audits and site maps.
- Platform:
- Screaming Frog
- User agent:
- screaming frog seo spider
- Tags:
- SEO
SEMrush crawler for site audits and technical checks.
- Platform:
- SEMrush
- User agent:
- siteauditbot
- Tags:
- SEO
SEMrush backlink audit crawler for link profiles.
- Platform:
- SEMrush
- User agent:
- semrushbot-ba
- Tags:
- SEO
On-page SEO checker crawler.
- Platform:
- SEMrush
- User agent:
- semrushbot-si
- Tags:
- SEO
SEMrush site-wide audit crawler.
- Platform:
- SEMrush
- User agent:
- semrushbot-swa
- Tags:
- SEO
SEMrush SplitSignal experimentation crawler.
- Platform:
- SEMrush
- User agent:
- splitsignalbot
- Tags:
- SEO
Content toolkit crawler for content optimization.
- Platform:
- SEMrush
- User agent:
- semrushbot-ocob
- Tags:
- SEO
Plagiarism checking crawler.
- Platform:
- SEMrush
- User agent:
- semrushbot-ft
- Tags:
- SEO
SEMrush Ryte integration crawler.
- Platform:
- SEMrush
- User agent:
- rytebot
- Tags:
- SEO
Enterprise crawler for large-scale SEO data.
- Platform:
- SEMrush
- User agent:
- semrushbot-es
- Tags:
- SEO
ByteDance crawler used internally and for social media.
- Platform:
- ByteDance
- User agent:
- twilio knowledge
- Tags:
- AI Training
Yahoo legacy crawler for web indexing.
- Platform:
- Yahoo
- User agent:
- slurp
- Tags:
- Search Engine Crawler
Yandex crawler for Russian search indexing.
- Platform:
- Yandex
- User agent:
- yandexbot
- Tags:
- Search Engine Crawler
Sogou search crawler.
- Platform:
- Sogou
- User agent:
- yisouspider
- Tags:
- Search Engine Crawler
You.com agent fetching pages at user request for AI search answers.
- Platform:
- You.com
- User agent:
- youbot
- Tags:
- AI User Initiated
Meta agent fetching content triggered by users.
- Platform:
- Meta
- User agent:
- meta-externalfetcher
- Tags:
- User Initiated (Non-AI)
Meta crawler that indexes public web pages.
- Platform:
- Meta
- User agent:
- meta-webindexer
- Tags:
- AI Search Index
Meta agent used for AI training data collection.
- Platform:
- Meta
- User agent:
- meta-externalagent
- Tags:
- AI Training
User-initiated fetches from Mistral sessions.
- Platform:
- Mistral AI
- User agent:
- mistralai-user
- Tags:
- AI User Initiated
