The “are you a robot” check used to block bad traffic can sometimes result in pages being removed from Google. In the last episode of Unofficial researchGoogle’s John Mueller explained why this happens, noting that it could cause Google to view your page as a duplicate of another site’s page.
The problem
When a site’s security flags a visitor as suspicious, it displays an “are you a robot” page instead of the actual content. Sometimes this page is returned to Google as if it were normal, causing Google to index it. This may result in the site’s actual content being removed from the index or replaced with this interstitial.
Since the “are you a robot” screen appears on other sites, Google encounters several similar pages. When it detects duplicate pages, Google selects one as the canonical version and considers the others as duplicates.
In addition to affecting indexing, this issue could also affect the page that Google considers to be the primary version of your content. It may select a page from another website, which will result in your page being marked as a duplicate.
Mueller explained that it is difficult to trace because it requires looking at the page selected by Google and then working backwards to determine the cause. Additionally, since a typical visitor may never encounter the question “are you a robot”, there doesn’t seem to be anything wrong when you examine the site yourself.
Why is it hard to see
Normal browser checks may not allow you to see the “are you a robot” screen, as it only appears for visitors flagged as suspicious. For you, the page probably loads normally.
You will need to go to Search Console to confirm there is a problem. The Page Index report marks pages as duplicates or as canonicalized elsewhere.
Additionally, Search Console’s URL inspection tool reveals which address Google has selected as the primary version. If this address belongs to a site that is not yours, it is a good idea to check it.
Why it matters
Sometimes issues can arise without any obvious signs on the site itself, as the “are you a bot” screen may be triggered by a CDN, host, or bot protection layer, rather than directly by the pages. Mueller mentioned that this type of protection can activate when exploration resumes.
The request is processed successfully, so the common instinct to search for a broken page is not helpful here. Since Google is able to access the site and get a valid response, the problem is that it is receiving the wrong content. This is why the failure is not detected by checks that do not recognize what Googlebot actually sees.
I addressed a similar problem where Mueller describes the “Indexed page without content” error. In this case, a website’s security settings silently blocked Googlebot but allowed repeat visitors, causing Google to load the page without receiving any content.
Looking to the future
If you encounter this problem, contact the person who manages your security service, CDN, or web hosting to find a solution. After fixing it, ask Google to recrawl via Search Consoles Validate Fix. Google may also detect the fix the next time it crawls.
Featured Image: Rock Tenys/Shutterstock
