Google-Agent

What does Google-Agent do?

Google-Agent navigates web pages and performs actions on behalf of users through Google-hosted AI agent products like Project Mariner. It fetches pages only when triggered by a user or another agent, not as part of continuous crawling. Because these agent products may cite or link to source pages in their responses, allowing Google-Agent can drive referral traffic back to your site.

Should I allow and optimize for Google-Agent to drive organic growth?

Google-Agent powers user-facing AI agent products like Project Mariner. When these products fetch your pages and present derived information, they may cite or link back to your content. This creates a direct referral traffic path from Google's AI ecosystem to your site. Because Google-Agent ignores robots.txt for user-triggered fetches, blocking it requires network-level controls. If you want visibility in Google's AI agent products, ensure your pages are accessible and well-structured. The indirect value is also significant: your content informing Google's agent responses keeps your brand present in AI-assisted workflows.

Data Usage & Training

It is unclear whether content fetched by Google-Agent is used for AI model training. Google's documentation does not explicitly address training use for this fetcher. Google provides a separate token, Google-Extended, for controlling whether your content is used for training and grounding purposes. If training use is a concern, configure Google-Extended in your robots.txt independently of Google-Agent.

How Google-Agent Accesses Content

Here's how Google-Agent accesses your site and understands your content:

• Fetches pages via standard HTTP requests using both mobile and desktop user-agent strings
• Triggered on-demand by end users or other agents, not by a scheduled crawl
• Identifies itself with a user-agent string containing 'Google-Agent' and a link to its documentation
• Source IPs are published in Google's JSON IP range files (e.g. user-triggered-agents.json at https://www.gstatic.com/ipranges/)
• Can be verified via reverse DNS lookup followed by forward confirmation of the hostname

Strictly on-demand. Google-Agent only makes requests when an end user or another agent triggers a fetch. There is no continuous or scheduled crawling.

How to Block or Control Google-Agent

Google-Agent generally ignores robots.txt Disallow directives because its fetches are user-initiated. Adding a robots.txt rule will not reliably block it: User-agent: Google-Agent Disallow: / For actual blocking, use IP-based controls. Google publishes IP ranges at https://www.gstatic.com/ipranges/ (check user-triggered-agents.json). You can verify requests by performing a reverse DNS lookup followed by forward confirmation. Apply firewall rules, CDN/WAF rules, or application-level controls (authentication, CAPTCHA) to block or challenge requests from those IP ranges.

Common Issues & Troubleshooting

Watch out for these common problems when working with Google-Agent:

• robots.txt Disallow rules are ineffective because Google-Agent ignores them for user-triggered fetches
• Relying on user-agent string alone for blocking is unreliable since UA strings can be spoofed
• CloudFlare or other WAF/bot protection may block legitimate Google-Agent requests by default
• Pages behind login walls or CAPTCHAs will be inaccessible to the agent
• JavaScript rendering behavior is undocumented, so heavily JS-dependent pages may not be fully processed