Google’s John Mueller recently responded to a question about A/B testing web pages for long durations, warning that an unintended consequence is that allowing variations to be indexed can lead to uncertainty about which ones will be visible in search results.
A/B testing traffic from live search results
A/B testing involves presenting one or more versions of a web page to users. The reason for this is usually to test conversion rates and user responses.
The important takeaway from the guidelines is that A/B testing of live web pages are guidelines that have been created to minimize the impact on search performance.
The guideline begins:
“This page explains how to ensure that testing variations in page content or page URLs has minimal impact on your Google search performance.”
Although Google does not explicitly prohibit the use of A/B testing to test which page ranks higher, the context of the guidelines themselves is framed as protecting search performance; measuring search performance is not part of the guidelines.
What Google’s document describes as being measured is consistently user behavior, not rankings.
By the way, what’s not in the guidelines is that there is no “right” button color or size to improve clicks on a call to action button. Long-time SEO knowledge and experience on this subject indicates that large buttons and/or colors that contrast strongly with the background of web pages tend to generate more clicks. This probably explains why Amazon’s Add to Cart button is a bright mustard color and Walmart’s version is a bright blue contrasting with a plain white background.
Google A/B Testing Guidelines
Google guidelines On A/B testing, they describe it as showing different versions of a website and collecting data on how users respond to them. In terms of SEO performance, you shouldn’t expect any disruptions, but by allowing Google to index slightly different pages after testing is complete, the winning combination will be indexed much sooner.
There are two types of A/B testing:
- A/B testing
Test two or more changes to a web page. Google uses the example of testing different fonts on buttons. - Multivariate tests
This is a test of several changes at the same time to identify which combination of factors work best together. Google uses the example of testing different combinations of different fonts on buttons and on the web page itself.
Four considerations for A/B testing
Google also recommends four best practices:
1. Use the rel=”canonical” link attribute
This is probably the most important factor to consider. Using the rel=canonical link attribute allows site owners to upload all kinds of variations of a web page while including a clear indication of which version of a web page is best.
2. Use 302 redirects
If you redirect users randomly to different versions of a web page, you should use a 302 redirectnot 301 redirects. 302 means that a resource (like a web page) has been temporarily moved. It’s different from a 301 redirect which means that a move or change of URL is permanent.
3. Don’t hide
Cloaking involves showing one thing to Google and another to users. If you test different web pages to see how users react when they click on a search, Google insists that site owners show Google the same thing, even if the elements on the page are constantly changing.
4. Don’t do A/B testing for a long time
Google warns site owners to limit the length of A/B testing. They warn that excessive testing could cause problems for a site:
“If we discover a site that is performing an unnecessarily long experience, we may interpret it as an attempt to trick search engines and take action accordingly. This is especially true if you serve a variation of content to a significant percentage of your users.”
This last warning directly concerns the question asked on the Bluesky social network.
Google answers a question about long-term A/B testing
The person asking the question specifically wanted to know how Google handles A/B testing that lasts up to a year.
They request:
“Hey @johnmu.com, as Google’s A/B testing guide suggests avoiding running the same A/B test for long durations, I was wondering how Google handles long-term resistance (e.g. 10% for 6-12 months), especially for a large-scale market with tens of millions of crawls to a similar number of pages.”
Google’s John Mueller responded:
“Depending on your setup, one or the other version may be used for indexing. If they are close enough, it probably doesn’t matter. If they are very different, it may also be visible in search results.”
The person who asked the original question then asked an additional question that revealed more about the extent of the changes in the web pages.
They asked:
“…what if it’s completely different, like a redesigned page, and since Googlebot gets alternate versions every crawl (sometimes within a day). Could this rapid change in the basic HTML structure cause indexing issues and potentially lead to Google removing pages from the index?”
Mueller responded:
“We take content into account in how we crawl it for indexing. There is no (to my knowledge) “penalty” or “demotion” for having variable content (many sites do), but it can make it harder to debug and monitor if the content is constantly changing.”
The questioner wanted to know how Google handles long-term A/B testing. They didn’t ask how Google handles indexing, but that’s the question Mueller answered. This may explain why the person asked a second, more specific question about the extent of their A/B testing and Mueller focused again on indexing.
No penalty for having variable content?
Mueller’s statement appears to contradict Google’s own advice regarding long-term A/B experiments.
The relevant context of Google’s guidelines is:
- This confirms that A/B testing is legitimate.
- It is reasonable to assume that normal experiences are temporary.
- Once enough data is collected to draw conclusions, the A/B test is expected to end.
This is where we come to the warning part of the guide:
“If we discover a site that is performing an unnecessarily long experience, we may interpret it as an attempt to trick search engines and take action accordingly. This is especially true if you serve a variation of content to a significant percentage of your users.”
So the point where things get shady is when the experience lasts longer than seems reasonable and one variation of the content becomes the primary version for most users in an attempt to “fool the search engines.»
Featured image by Shutterstock/logofank
