OAI-SearchBot

What does OAI-SearchBot do?

OAI-SearchBot is OpenAI's crawler for indexing web pages to power ChatGPT's search features (including ChatGPT Atlas). It surfaces page titles and citation links directly in ChatGPT search results, distinct from GPTBot, which handles training-related crawling. Citations include a utm_source=chatgpt.com parameter, so you can track referral traffic in your analytics.

Should I allow and optimize for OAI-SearchBot to drive organic growth?

OAI-SearchBot directly powers ChatGPT search, which surfaces clickable citation links back to your site. These citations include UTM parameters (utm_source=chatgpt.com), making referral traffic easy to measure. Allowing this crawler is one of the most direct ways to gain visibility in ChatGPT's search results. Blocking it removes your page content from those results, though OpenAI may still surface a link and title obtained from third-party providers.

Data Usage & Training

Content crawled by OAI-SearchBot is not used for AI model training. OpenAI uses a separate crawler, GPTBot, for training-related crawling. To opt out of training, block GPTBot in your robots.txt. Blocking OAI-SearchBot only affects whether your content appears in ChatGPT search results.

How OAI-SearchBot Accesses Content

Here's how OAI-SearchBot accesses your site and understands your content:

• Fetches HTML via standard HTTP requests using a Chrome-based user-agent string
• Does not render JavaScript
• Respects robots.txt Allow and Disallow directives for the OAI-SearchBot token
• Respects meta noindex tags (but only if the crawler is allowed to fetch the page)
• Source IPs can be verified against OpenAI's published IP list at https://openai.com/searchbot.json

OAI-SearchBot performs regular, continuous indexing rather than purely on-demand fetches. Robots.txt changes can take approximately 24 hours to propagate to search results.

How to Block or Control OAI-SearchBot

To block OAI-SearchBot from indexing your content for ChatGPT search: User-agent: OAI-SearchBot Disallow: / You can also use meta noindex on specific pages, but the crawler must be allowed to fetch the page to read the tag. For IP-based blocking, verify requests against OpenAI's published IP list at https://openai.com/searchbot.json. Blocking OAI-SearchBot does not affect training. To opt out of training, block GPTBot separately.

Common Issues & Troubleshooting

Watch out for these common problems when working with OAI-SearchBot:

• Robots.txt changes can take up to 24 hours to propagate to ChatGPT search results
• Blocking the crawler via robots.txt prevents it from reading meta noindex tags, so the tag has no effect on pages it cannot fetch
• Confusing OAI-SearchBot with GPTBot leads to incorrect robots.txt rules; use the exact token for each
• JavaScript-rendered content is not accessible to this crawler since it does not execute JavaScript
• Even when blocked, OpenAI may still surface a page's link and title obtained from third-party data providers