Google is testing Web Bot Auth, an experimental protocol designed to help websites verify that automated traffic actually comes from the bot or service it claims to represent. The new protocol could provide site owners with a reliable way to separate legitimate automated traffic from bots that hide or misrepresent their identities.
A new developer support page has been released to provide information on how to verify requests with the Web Bot Auth protocol, which is currently in an experimental phase.
What is Google Web Bot authentication based on?
The new protocol is technically called the HTTP Message Signature Directory. It is a proposed technical standard designed to automate trust between web services. It helps websites recognize verified automated services without requiring each party to manually exchange security keys first.
The basic idea comes down to giving verified automated services a standardized way of presenting credentials. Instead of relying solely on names, user agent strings, or private configuration between companies, the protocol gives websites a repeatable way to check whether an automated request can be verified. This is important because many bots may pretend to be something they are not. Web Bot Auth doesn’t decide whether a bot is good or bad, but it can give site owners a stronger signal about whether the bot is actually the service it claims to be.
A reliable way to identify bots
The cryptographic part is important because it makes the identity more difficult to forge. Today, a malicious bot can pretend to be a legitimate crawler by copying a name or user agent string. Web Bot Auth is designed to go beyond this type of self-identification by giving websites a way to check whether an automated request matches the service’s cryptographic credentials.
Under this protocol, a robot would need more than a label indicating who it is. This identity would have to be proven in a way that a website could validate. This could give site owners a secure basis for allowing verified automated services while blocking bots that can’t prove who they are. The protocol doesn’t automatically decide which bots should be allowed or blocked, but it could give websites a more reliable signal to make that decision.
Cryptographic verification is what makes Web Bot Auth better than current bot identification methods. Instead of relying on signals that can be distorted, this gives websites a way to verify automated requests. This means that recognition relies less on what a robot says about itself and more on whether its identity can be confirmed by cryptographic credentials.
Please note: this is in an experimental phase
The proposed protocol will make it possible to distinguish malicious robots that pose as trusted robots from real robots from trusted services. This protocol is like a whitelist of what is allowed, which can make it easier to isolate untrusted crawlers.
However, as this is an experimental phase, “whitelisting” currently only applies to a subset of traffic, such as Google-Agent. Google “does not yet sign all requests,” so a missing signature does not automatically mean a bot is malicious. Site owners are advised to continue to use IP addresses and reverse DNS alongside the protocol to avoid accidentally blocking legitimate traffic that has not yet migrated.
What he does
The new standard replaces manual configuration between websites and bots, crawlers, and other automated services with a three-step discovery process:
- Standardized key files:
Keys are stored in a common format, JSON Web Key Set (JWKS), that all servers can read. - Known addresses:
It defines a specific “home” on a website (/.well-known/) where these keys are always kept. - Self-identification requests:
It adds a new header, Signature-Agent, to HTTP requests that acts like a digital business card, pointing the recipient directly to the sender’s key directory.
Benefits for automated services and websites
Web Bot Auth could make bot verification easier to scale by reducing the need for manual configuration between each website and automated service. It also gives automated services a more consistent way to remain recognizable when their security information changes, which can help avoid broken verification over time.
Web bot authentication is experimental
Google emphasizes that users should continue to use existing standards such as bot verification based on the IP address of user agents, emphasizing that the standard itself is a proposal subject to change.
The new documentation provides the following warning:
“Experimental status means that:
Not all Google user agents use Web Bot Auth.
Google does not yet sign all requests from agents using the protocol.
We recommend that, in addition to Web Bot Auth, you continue to rely on IP addresses, reverse DNS, and user agent strings as we gradually roll out signed traffic.
If you are a developer or system administrator interested in authorizing our experimental AI agents, you can implement verification via the Web Bot Auth protocol:
- Using a product or service that supports Web Bot Auth
- Check the requests yourself”
Nonetheless, the standard aims to make it simpler to identify bots and control their traffic using a cryptographic protocol that a malicious agent cannot spoof, provide insight into how bots interact with your traffic, and create a better way to control the currently out-of-control situation with bot crawling.
Google encourages users interested in the protocol to contact their web hosting providers to see if they intend to support the experimental protocol, stay up to date with the latest changes published by the Web Bot Auth Working Group, and submit feedback via Google’s official Web Bot Authentication feedback form.
Read the new Google documentation:
Authenticate requests with Web Bot Auth (experimental)
Featured image by Shutterstock/Efkaysim
