Google’s Martin Splitt and John Mueller discussed the Page Index Report and how it could be used to identify indexing issues. It’s interesting how they linked quality to indexing, saying that quality isn’t just about the text on a web page.
Indexing is done in stages
Splitt and Mueller were discussing issues that might temporarily affect indexing, such as an unresponsive server, computers failing, DNS lookups sometimes failing, many reasons why Google might not be able to crawl a site, and then gave the advice that not everything needs to be investigated.
Then they mentioned that when a site is new, the Page Index Report can show you the different indexing stages a page goes through.
Splitt explained how it works:
“And also, if you’re adding or changing your site, or if your site is very new, you can also use this report to kind of see how your site goes through the different stages.
Because at some point you will see discovered pages that are currently not indexed, which tells you that we know they exist, but we haven’t actually visited them. And if we haven’t visited them, we can’t put them in the index while the crawled ones are currently not indexed, which means we visited them and we didn’t put them in the index. And that can have all kinds of different reasons.
Lack of indexing is not always a quality problem
At this point the conversation turned to quality issues in the context of indexing and how this might impact. Mueller confirmed that it’s not always about quality.
Splitt asked Mueller about the unindexed status in the Page Index report:
“Would you say this is often or only sometimes a sign of a quality problem? »
Müller replied:
“Sometimes.”
Strong concerns about quality
Mueller explained that unindexed status can be triggered if Google’s systems have “serious concerns” about quality. It seems strange that a system would be in a state of worry. What he probably meant is that if quality scores or another indicator exceed a threshold, it may mean that there is a “concern” about quality related to the site.
Mueller continued his response:
“So certainly if our systems have serious concerns about the quality of the website, they will reduce the number of pages in the index. Because if we have serious concerns about the overall quality, it doesn’t make much sense for our systems to spend a lot of time on the website.”
So we’ll probably crawl a lot less. We will index much less. And then you’ll see things like explored unindexed or discovered unindexed, which from our perspective is basically our system saying, we’re aware and once we’re happy with it, we’ll take another look at it and see if we can index it.
Page not indexed not always for a technical reason
It is not uncommon for a site owner to assume that there is nothing wrong with their site, leading to an inability to diagnose a page quality issue. No one knows a site as well as the site owner or SEO, and they have taken the time to make it as perfect as possible. So it’s understandable that someone might not be able to see that there are real quality issues affecting a site.
What happens is people try to find a technical problem to explain why a site isn’t indexed. When this fails, some people think they are victims of negative SEO and begin a cycle of constantly disavowing “bad” links, usually without improvement.
If this sounds like your situation, you may need to seriously consider that there is a quality issue affecting the site, which may seem inconceivable because the site is “perfect”.
Mueller addresses this quality blind spot that can affect any site owner.
Mueller continued his response:
“It’s not so much that I would say you should take these situations and try to resolve them. From a technical standpoint, it’s not that you need to resolve this technical issue because Google isn’t indexing this page right now. But rather, you almost need to… when you recognize a broader pattern like this, like Google isn’t indexing many of your pages, and there’s no technical reason, you almost have to step back and think about overall quality.
And thinking about quality is really challenging because a lot of times it’s your website and your baby. And of course, he’s the best baby ever.
Site visitors and overall quality
At this point, Mueller adds something unexpected to the discussion. He mentions how site visitors can take a look at the site and quickly determine that it lacks quality factors such as uniqueness. He didn’t say that Google’s systems determined that the site lacked quality; he said maybe it’s the people who will make that decision.
It did not explicitly state that Google was picking up on user signals indicating site quality issues. But it looks like he’s implying it.
Mueller continued his response:
“But taking a step back and trying to look at it with the eyes of someone who isn’t directly involved with your website, sometimes that opens up ideas about areas where you can improve, where maybe if the majority of your website is AI-generated and it’s been running for a while, it might be that people look at that AI-generated site and say, well, I can say it’s AI-generated, there’s nothing something unique or valuable that is available here for me.
That’s not to say that all AI-generated content is bad, but sometimes you just browse websites where you think, anyone could have written this. That doesn’t mean anything to me.
Martin Splitt highlighted how difficult it is to recognize that your best efforts may not be as perfect as they seem.
Split said:
“Yeah, that’s true.
And I think what makes this difficult is not just the fact that the way you wrote it is obviously the way you thought was best, and that’s why you think it’s high quality, of course. So it’s really very difficult to get out of your own point of view.
But sometimes there are also so many other things that are just as good. So why would we add it to the index?
And then that can tell you, maybe this content isn’t as valuable as I thought it was, it’s because other people are covering the same thing. And then, what is the value of this version in the index? »
Overall quality isn’t just about text
Mueller responded to Splitt by mentioning that quality isn’t always about the text.
“I think we could have a whole podcast about quality. I think another thing that’s worth mentioning when it comes to quality is that it’s not just about the text. So a lot of times people will say, well, my writing is unique or my articles are good.
And they’re bundled into a hard to access page where anyone, when they try to load it, like their computer fan is spinning and they’re like, oh my God, I need to run away to make sure my computer doesn’t explode. So maybe this is an extreme case.
If you only have one takeaway from this podcast, it should be this: Mueller says quality isn’t just about the text. He said the way a user experiences a web page can also be part of the quality.
User Experience and Quality Issues
Mueller gave examples of the types of non-textual quality issues that can negatively impact user experience. He didn’t say these issues could cause a person to abandon the site, but the examples he shared are the kinds of things that can cause a site visitor to hit the back button to leave the web page.
This is interesting because it’s the kind of thing that can be detected, a signal of user behavior. He used the example of a recipe site that frustrates visitors to its site by making it difficult to find the recipe. This is an example of a poor user experience as evidence of a quality problem.
Mueller actually calls user experience what it is, saying they almost have to consider “the whole experience” for quality.
He continued:
“But you’ve all seen those pages where the text is basically there, but it’s almost hidden, hidden behind ads, hidden behind interstitials, hidden behind other things that move and go back and forth, maybe hidden under a bunch of filler content, which we sometimes see, for example, with recipes where there’s this really long story at the top that maybe most people don’t really care about, and then the recipe comes.
This is all the kind of stuff where the overall quality is much more than this piece of text that you say, like this is my main content, this is what Google should count for my site.
And from our perspective, we almost have to consider the full experience on a page because that’s what users see.
It’s not about users going to a web page and turning on some magical mode that just extracts the text, but rather they’re getting the full experience of that website with all the 3D, 4D animations and everything.
Takeaways
- Google may reduce crawling and indexing when its systems have strong concerns about the overall quality of a website.
- Site owners and SEOs should investigate quality issues when pages are not indexed and there are no technical issues to blame.
- Quality depends partly on whether the pages offer unique and useful information. In my opinion, unique does not mean that the words are literally different from another site. Unique means the topic, cover, all elements of the content and the way it is presented to the user.
- Slop content, both AI-generated and human-generated, can trigger a quality issue that manifests itself in site visitors abandoning the page.
- Google considers the entire page experience, including accessibility, performance, ads, interstitials, filler content, and how easily users can access the information they need.
Google’s John Mueller explained that widespread indexing issues can reflect concerns about the overall quality of a site. He suggested that user experience is a factor that can influence indexing. The point to consider is that technical issues may not be the reason a site is not indexed and text is not always the reason either, which has to do with user experience.
Featured image by Shutterstock/JHVEPhoto
