ShapBot

What does ShapBot do?

ShapBot discovers and indexes websites for Parallel's web APIs and AI search indexing services. It collects pages and IP-level signals to populate Parallel's search infrastructure. There is no documented mechanism for driving referral traffic or citations back to your site.

Should I allow and optimize for ShapBot to drive organic growth?

ShapBot feeds Parallel's web APIs and AI search index, which other products and developers may consume. While there is no documented referral traffic or citation mechanism, having your content indexed in Parallel's infrastructure means it could surface through downstream applications built on their APIs. Allowing ShapBot is a reasonable default if you want broad visibility across AI-powered search products.

Data Usage & Training

It is unclear whether content crawled by ShapBot is used to train AI models. Parallel's documentation describes indexing for web APIs but does not explicitly address model training. If this distinction matters to you, contact [email protected] for clarification.

How ShapBot Accesses Content

Here's how ShapBot accesses your site and understands your content:

• Fetches pages via standard HTTP requests using the user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0
• Collects IP-level signals alongside page content
• Reads Sitemap directives from robots.txt
• Respects Allow and Disallow rules in robots.txt
• JavaScript rendering capability is unknown

Not publicly documented. ShapBot likely crawls on a continuous, recurring basis given its role as a search index crawler, but no specific schedule or rate limits have been published.

How to Block or Control ShapBot

To block ShapBot via robots.txt: User-agent: ShapBot Disallow: / For IP-based blocking, Parallel publishes an IP range list at https://docs.parallel.ai/resources/shapbot.json. Use this list to build accurate allow or deny rules in your firewall or CDN. You can also contact [email protected] to request removal or opt out. No public opt-out form exists.

Common Issues & Troubleshooting

Watch out for these common problems when working with ShapBot:

• ShapBot operates from many IP addresses, making UA-only blocking insufficient if you need strict control
• Some site owners have misclassified ShapBot IP ranges as malicious; verify against the published shapbot.json list before blocking
• Crawl-delay is not documented as supported, so rate limiting via robots.txt may not work
• JavaScript-rendered content may not be fully indexed, as rendering capability is unknown