Google’s John Mueller answered a question about the robots.txt file and explained an easy-to-miss error that can impact your website SEO and indexing goals. The specific problem was related to search box spamming indexed by Google, but this error can happen to anyone in general in any context.
Website search box spam
The person who asked the question on Reddit was suffering from a spam attack in the search bar. What spammers do is perform a search with a query that reflects their spam niche, and they add a link or website name. What happens next is that the search bar generates a URL that can be referenced to generate the spammy search result.
And that’s what happened to the person asking the question. Their response was to add a line to the robots.txt file to prevent Google from indexing the file. But Google indexed these URLs generated by spammy searches anyway.
Google indexed pages blocked by Robots.txt
Someone posted on Reddit that their client’s Shopify search box was generating spammy web pages in response to spammers’ queries and that Google was indexing them despite a robots.txt file prohibiting Google from indexing those pages. The client redirected these spammy URLs to another web page. The person who asked the question didn’t ask how to prevent pages from being indexed (which they should have asked); they asked if these redirected URLs should be marked 404 instead.
The person request:
“I’m working on a client’s Shopify store where the /search is added as a ban in robots.txt, but those search results are still indexed in Google.
However, if I try to open any of these pages, they have a redirect set and they redirect to another collection page in the store. Should we instead display a 404 page? What is the simplest way to solve this problem? »
Why Robots.txt file caused spam indexing
Google’s John Mueller took the extra step to identify and review the client’s robots.txt file and identified an error that caused Google to ignore the directive prohibiting Googlebot from indexing search results pages.
Mueller responded:
“Also, I don’t know if this is your site, but the one I found with similar indexed URLs had sections for “user-agent: Googlebot” (in the “START: Custom Rules” block in the comments) as well as a lot more in the “user-agent: *” section further down. With robots.txt, the most specific rules win, so if you have a user-agent: Googlebot section, it will *only* use that section. you want to apply all the rules from “user-agent:*”, you need to copy them. Additionally, if this is your site, you can simply list all the user-agents for which you want to share rules together, for example:
user agent: googlebot
user agent: other robot
user agent: imgsrc
user agent: something
prohibit: /fish
ban: /orange-cats
… etc …”
User agent specific directives take precedence
What happened was that the client was counting on Google to follow the directives of one line intended for all user agents, “user-agent: *”, but because there is another section of the robots.txt file specific to Googlebot, Google ignored the “user-agent: *” directives and obeyed the one that was directed specifically to Googlebot.
This may seem like a quirk in how robots.txt works, but it makes sense because it allows users to target specific crawlers with unique rules and target all others with a different set of rules.
How to protect yourself from search box spam
Both WordPress and Shopify have ways to mitigate search box spam.
Both WordPress and Shopify have ways to mitigate search box spam.
Mitigation of Shopify search box spam
The Shopify website has a tutorial on how to automatically add a noindex directive to all search results. This will effectively prevent all search results pages from being indexed. However, it is necessary to not block search pages with robots.txt for this to work.
Using a noindex directive is more efficient than using robots.txt because robots.txt does not control indexing; it only controls exploration.
Shopify educated:
“You can hide pages that are not included in your robots.txt.liquid file by customizing the
section of your store’s theme.liquid layout file. You must include meta tag code to stop indexing of particular pages.
From your Shopify admin, navigate to Online Store > Themes.
Find the theme you want to edit, click the … button to open the actions menu, and then click Edit Code.
In the layout folder, click on the theme.liquid file.
To exclude the search pattern, paste the following code on a blank line in the field
section:
{% if the model contains ‘search’ %}
{% endif %} »
Mitigating Spam in the WordPress Search Box
It’s pretty easy to mitigate search box spam with WordPress. Users of the Yoast, Rank Math, and AIOSEO SEO plugins have their search results pages automatically set to noindex by default. Additionally, some page builders and themes like Divi (and its Extra theme) will not automatically generate spammy words or URLs in response to searches and will instead inject a few sentences indicating that the search produced no results.
Knowledge of the Robots.txt file
Effective SEO requires a wide range of knowledge. Robots.txt contains a few quirks that may make it less efficient than expected, so it’s worth reading the official specifications in order to stay up to date.
Featured image by Shutterstock/Stockinq
