March 20, 2026, Google quietly added a new entry to its official list of web scrapers. Not a robot. Not a training robot. An agent.
Google-Agent is the user agent string for AI systems running on Google infrastructure that crawl websites on behalf of users. When someone asks an AI assistant to search for a product, fill out a form, or compare options across websites, it’s the Google agent who actually visits the page. Marine ProjectGoogle’s experimental AI navigation tool, is the first product to use it.
It’s not Googlebot. Googlebot continuously explores the Web, indexing pages for search. Google-Agent only appears when a human asks it to. This distinction changes everything in its operation.
Robots.txt does not apply
Google classifies Google-Agent as a user triggered recovery tool. The category includes tools such as Google Read Aloud (text-to-speech), NotebookLM (document analysis), and Feedfetcher (RSS). They all share one property: a human initiated the request. Google’s position is that user-triggered fetchers “generally ignore the rules in robots.txt” because the fetch was requested by a person.
The logic: If you enter a URL in Chrome, the browser retrieves the page regardless of what the robots.txt file says. Google-Agent works on the same principle. The agent is the user’s proxy and not an autonomous crawler.
This is a significant change from how OpenAI and Anthropic handle similar traffic. ChatGPT-User And Claude-User both work as user-triggered grabbers, but they respect robots.txt directives. If you block ChatGPT-User in robots.txt, ChatGPT will not fetch your page when a user asks it to navigate. Google made a different call.
Website owners who relied on robots.txt as a universal access control mechanism, there is now a gap. If you need to restrict access from Google-Agent, you will need server-side authentication or access controls. The same tools you would use to block a human visitor.
Cryptographic identity: web bot authentication
The most important development is buried in a single line of Google text documentation: Google-Agent is experimenting with web-bot-auth protocol using identity https://agent.bot.goog.
Web bot authentication is a draft IETF standard that functions as a digital passport for robots. Each agent holds a private key, publishes its public key in a directory, and cryptographically signs each HTTP request. The website verifies the signature and knows, with cryptographic certainty, that the visitor is who they say they are.
User agent strings can be spoofed by anyone. Web Bot authentication cannot. Google’s adoption of this protocol, even as an experiment, indicates where agent identity is headed. AkamaiCloudflare and Amazon (AgentCore Browser) already support it. Google provides critical mass.
This is important because the web is on the verge of an identity problem. As agent traffic increases, websites must distinguish between legitimate AI agents acting on behalf of real users and scrapers posing as agents. IP verification is useful, but cryptographic signatures scale better and are harder to forge.
What this means for your website
Google-Agent creates a three-tier visitor model for the web:
- Human visitors navigate directly.
- Crawlers content indexing for research and training (Googlebot, GPTBot, Google-Extended).
- Agents act on behalf of specific humans in real time (Google-Agent, ChatGPT-User, Claude-User).
Each level has different access rules, different intentions and different expectations. A robot wants to index your content. An agent wants to complete a task. This could be reading a product page, comparing prices, filling out a contact form or making an appointment.
Here’s what to do now:
Monitor your logs. Google-Agent identifies itself with a user-agent string containing compatible; Google-Agent. Google publishes IP ranges for verification. Start tracking how often agents visit, what pages they view, and what they are trying to do.
Check your CDN and firewall rules. If your security tools aggressively block non-browser traffic, Google Agent may be rejected before it reaches your server. Verify that the IP address ranges published by Google are allowed.
Test your forms and flows. Google-Agent can submit forms and navigate multi-step processes. If your payment, booking, or contact forms rely on JavaScript templates that confuse automated systems, agent visitors will fail silently. Semantic HTML and clear labels remain the foundation.
Accept that robots.txt is no longer a comprehensive access control tool. For content you actually need to restrict, use authentication. robots.txt was designed for crawlers. The Age of Agents requires different boundaries.
The hybrid web is not happening. It’s recorded
A year ago, the idea that AI agents would navigate websites alongside humans was just a prediction at a conference. Today it has a user agent string, published IP address ranges, a cryptographic identity protocol, and an entry in Google’s official documentation.
The Web has not divided into humans and machines. He merged. Every page you publish now serves both audiences simultaneously, and Google has just made it possible to see exactly when the non-human audience appears.
More resources:
This article was originally published on No hacks.
Featured image: Summit Art Creations/Shutterstock
