GPTBot

What does GPTBot do?

GPTBot crawls public web content to help train and improve OpenAI's generative AI foundation models. It feeds data into OpenAI's model training pipeline, not into user-facing search features. GPTBot does not drive referral traffic or citations back to your site. OpenAI uses separate crawlers (like OAI-SearchBot) for ChatGPT's search features, which do include citation links.

Should I allow and optimize for GPTBot to drive organic growth?

GPTBot itself does not generate referral traffic or citations. However, allowing it contributes training data to OpenAI's foundation models, which power ChatGPT and other OpenAI products used by hundreds of millions of people. Content that informs model training can indirectly influence the quality and accuracy of AI-generated responses about your brand, products, or domain. Blocking GPTBot removes your content from that training pipeline entirely. If visibility in AI-powered products matters to you, allowing GPTBot is worth considering, even though the growth mechanism is indirect.

Data Usage & Training

Content crawled by GPTBot may be used to train OpenAI's generative AI foundation models. Blocking GPTBot in your robots.txt signals to OpenAI that your site should not be used for training. Changes to robots.txt can take roughly 24 hours to be picked up by the crawler.

How GPTBot Accesses Content

Here's how GPTBot accesses your site and understands your content:

• Fetches HTML via standard HTTP requests
• Does not render JavaScript
• Identifies as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot
• Crawls from published IP ranges listed at https://openai.com/gptbot.json
• Respects robots.txt Allow and Disallow directives for the GPTBot token
• Respects meta robots noindex tags when able to fetch the page

GPTBot crawls on a periodic, regularly scheduled cadence rather than on-demand. Robots.txt changes can take approximately 24 hours to be picked up.

How to Block or Control GPTBot

To block GPTBot from crawling your site, add this to your robots.txt: User-agent: GPTBot Disallow: / To block specific sections while allowing others: User-agent: GPTBot Allow: /public/ Disallow: /private/ For IP-based blocking, OpenAI publishes GPTBot's IP ranges at https://openai.com/gptbot.json. You can use these in your firewall or CDN/WAF rules. You can also add WAF rules targeting the GPTBot user-agent string. Robots.txt changes may take up to 24 hours to take effect. Blocking GPTBot does not affect OAI-SearchBot, so your site can still appear in ChatGPT search results.

Common Issues & Troubleshooting

Watch out for these common problems when working with GPTBot:

• Robots.txt changes take up to 24 hours to propagate, so blocking won't take effect immediately
• Confusing GPTBot with OAI-SearchBot is common; blocking GPTBot only stops training crawls, not ChatGPT search crawls
• Sites behind CDNs or proxies may see masked source IPs, making IP verification harder; cross-reference with the published IP list at https://openai.com/gptbot.json
• Meta robots noindex tags only work if GPTBot is allowed to fetch the page first; a robots.txt Disallow prevents the crawler from ever seeing the tag
• GPTBot does not render JavaScript, so content loaded dynamically via client-side frameworks will not be crawled