Directory of Agents & Bots for AI Search Optimization
Browse public profiles for known AI agents, crawlers and search bots. Learn what they do,
how they access content and which platforms they belong to.
Ahrefs crawler used for backlink and SEO analysis.
- Platform:
- Ahrefs
- User agent:
- ahrefsbot
- Tags:
- SEO
Ahrefs site auditing bot that checks technical SEO issues.
- Platform:
- Ahrefs
- User agent:
- ahrefssiteaudit
- Tags:
- SEO
Allen Institute crawler for AI training datasets.
- Platform:
- AI2
- User agent:
- ai2bot
- Tags:
- AI Training
Enterprise search indexing crawler for Amazon Kendra.
- Platform:
- Amazon
- User agent:
- amazon-kendra
- Tags:
- Search Engine Crawler
Amazon Q assistant fetching content to answer user queries.
- Platform:
- Amazon
- User agent:
- amazon-qbusiness
- Tags:
- AI Training
Amazon crawling agent used for AI training and discovery.
- Platform:
- Amazon
- User agent:
- amazonbot
- Tags:
- AI Training
Crawler for product discovery and search indexing.
- Platform:
- Amazon
- User agent:
- amazonproductdiscoverybot
- Tags:
- E-commerce, Search Engine Crawler
Google service discovery bot for API-related resources.
- Platform:
- Google
- User agent:
- apis-google
- Tags:
- Search Engine Crawler
Apple web crawler for indexing and AI training.
- Platform:
- Apple
- User agent:
- applebot
- Tags:
- Search Engine Crawler, AI Training
Baidu's primary crawler for web search indexing.
- Platform:
- Baidu
- User agent:
- baiduspider
- Tags:
- Search Engine Crawler
Qwant's SEO crawler for site analysis and indexing.
- Platform:
- Qwant
- User agent:
- barkrowler
- Tags:
- SEO
Microsoft Bing web crawler for search indexing.
- Platform:
- Microsoft
- User agent:
- bingbot
- Tags:
- Search Engine Crawler
ByteDance data collection bot used for AI training.
- Platform:
- ByteDance
- User agent:
- bytespider
- Tags:
- AI Training
Open web crawl used widely in AI training datasets.
- Platform:
- Common Crawl
- User agent:
- ccbot
- Tags:
- AI Training
OpenAI browsing agent fetching pages at user request.
- Platform:
- OpenAI
- User agent:
- chatgpt-user
- Tags:
- AI User Initiated
Microsoft Clarity bot for SEO and verification checks.
- Platform:
- Microsoft
- User agent:
- claritybot
- Tags:
- SEO
Anthropic crawler that indexes content for Claude search.
- Platform:
- Anthropic
- User agent:
- claude-searchbot
- Tags:
- AI Search Index
User-initiated fetches triggered by Claude sessions.
- Platform:
- Anthropic
- User agent:
- claude-user
- Tags:
- AI User Initiated
Anthropic crawler for collecting training data.
- Platform:
- Anthropic
- User agent:
- claudebot
- Tags:
- AI Training
DataForSEO bot collecting ranking and SERP data.
- Platform:
- DataForSEO
- User agent:
- dataforseobot
- Tags:
- SEO
Moz crawler for link analysis and SEO metrics.
- Platform:
- Moz
- User agent:
- dotbot
- Tags:
- SEO
DuckDuckGo assistant fetching content for answers.
- Platform:
- DuckDuckGo
- User agent:
- duckassistbot
- Tags:
- AI User Initiated
DuckDuckGo crawler for privacy-first web indexing.
- Platform:
- DuckDuckGo
- User agent:
- duckduckbot
- Tags:
- Search Engine Crawler
Google agent assisting users with deep research tasks.
- Platform:
- Google
- User agent:
- gemini-deep-research
- Tags:
- AI User Initiated
Vertex AI agent used to retrieve and index documents.
- Platform:
- Google
- User agent:
- google-cloudvertexbot
- Tags:
- AI User Initiated
Opt-in crawler for Google AI model training.
- Platform:
- Google
- User agent:
- google-extended
- Tags:
- AI Training
Accessibility agent that reads page content aloud.
- Platform:
- Google
- User agent:
- google-read-aloud
- Tags:
- User Initiated (Non-AI)
Google Store bot for product content and search.
- Platform:
- Google
- User agent:
- storebot-google
- Tags:
- E-commerce, Search Engine Crawler
Primary Google crawler for search indexing.
- Platform:
- Google
- User agent:
- googlebot
- Tags:
- Search Engine Crawler
Internal Google crawler used for non-public systems.
- Platform:
- Google
- User agent:
- googleother
- Tags:
-
OpenAI crawler used to gather training data for GPT.
- Platform:
- OpenAI
- User agent:
- gptbot
- Tags:
- AI Training
Naver search crawler used in Korean search index.
- Platform:
- Naver
- User agent:
- yeti
- Tags:
- Search Engine Crawler
OpenAI crawler indexing content for answer engines.
- Platform:
- OpenAI
- User agent:
- oai-searchbot
- Tags:
- AI Search Index
User-initiated fetches triggered from Perplexity sessions.
- Platform:
- Perplexity
- User agent:
- perplexity-user
- Tags:
- AI User Initiated
Perplexity crawler that indexes the public web.
- Platform:
- Perplexity
- User agent:
- perplexitybot
- Tags:
- AI Search Index
Desktop SEO spider used for audits and site maps.
- Platform:
- Screaming Frog
- User agent:
- screaming frog seo spider
- Tags:
- SEO
ByteDance crawler used internally and for social media.
- Platform:
- ByteDance
- User agent:
- twilio knowledge
- Tags:
- AI Training
Yahoo legacy crawler for web indexing.
- Platform:
- Yahoo
- User agent:
- slurp
- Tags:
- Search Engine Crawler
Yandex crawler for Russian search indexing.
- Platform:
- Yandex
- User agent:
- yandexbot
- Tags:
- Search Engine Crawler
Sogou search crawler.
- Platform:
- Sogou
- User agent:
- yisouspider
- Tags:
- Search Engine Crawler
