Google Dumps The Supplemental Results Label

As many expected, Google has announced that it is dumping the supplemental result label for pages listed within its supplemental index. Does that mean the supplemental index has gone away? No. But Google promises that being in the supplemental index versus the main index should be less of an issue for site owners, going forward. More about the change, below.

Back in September 2003, Google unveiled the supplemental index as a way for it to index more pages from the web but not get bogged down actually searching through those billions and billions of them unless necessary. By and large, it was less important pages that ended up in the supplemental index. If Google’s main index — containing what it considered the best or most important pages — didn’t seem to have a match, then Google would look to the supplemental index for listings.

Pages in the supplemental index could, did and do rank — but only generally if there’s not much to be found in the primary index. As a result, site owners who discovered that many of their pages were in the supplemental index would worry. They interpreted that as a sign that Google saw much of their content to be not as important. And that was a fair enough interpretation. The pages really were "second class" in the sense of being listed only if the main index had nothing gripping plus the fact that Google might not revisit pages in the supplemental index for months.

Concerned site owners have come up with various ways to tell if they have pages in the supplemental index, and one of these methods started failing to work earlier this month. Is Google Gearing Up To Drop The Supplemental Result Label? from us explains what happened more and how a comment from Google’s Matt Cutts prompted speculation that the label would go away.

Matt was indeed laying the groundwork for today’s news. From the Google Webmaster Central blog post, Supplemental goes mainstream:

Since 2006, we’ve completely overhauled the system that crawls and indexes supplemental results. The current system provides deeper and more continuous indexing. Additionally, we are indexing URLs with more parameters and are continuing to place fewer restrictions on the sites we crawl. As a result, Supplemental Results are fresher and more comprehensive than ever. We’re also working towards showing more Supplemental Results by ensuring that every query is able to search the supplemental index, and expect to roll this out over the course of the summer.

The distinction between the main and the supplemental index is therefore continuing to narrow. Given all the progress that we’ve been able to make so far, and thinking ahead to future improvements, we’ve decided to stop labeling these URLs as "Supplemental Results." Of course, you will continue to benefit from Google’s supplemental index being deeper and fresher.

Let’s deal with the labels first. As a reminder, in case you’ve never noticed them, here’s an example of how they’ve looked:

That’s my article from April about PageRank. See the words "Supplemental Result" on the third line? That tells you this page is listed in the supplemental index. But that label is going away. Like now. The process has already started, and it will take a few days to rollout completely across various Google data centers, Google told me. If you still see them, it won’t be for long.

At the moment, if you want to force the labels to show up, doing a search for [site:domain/&] is a tip that came out of WebmasterWorld this week, and that still seems to be still working. Search Engine Roundtable has more about it, and we’ve also had discussion on it at Sphinn here. You can see it in action with this search for Search Engine Land:

site:searchengineland.com/&

Ready to be angry and upset that Google is preventing you from knowing a key status indicator of your pages? Hang on, because there’s hope on two different fronts.

First, as the blog post explains, by the end of summer, there should be less difference between the supplemental index and the main one. Google is promising that every query should hit both indexes at the same time, so that a page in supplemental really should have a better chance of ranking than in the past. Google’s also said freshness has increased, so that no page in the supplemental index is more than three months out of date and that this might be reduced to only a month in the future.

Of course, saying freshness has increased to being only "three months old" is kind of a contradiction. It’s true that many pages in the supplemental index won’t change very often. But some do, so it would be good for Google to step up an event faster crawl of supplementals and kick more of them out into the main index, as warranted. The balancing act, of course, is you can have site owners making very minor changes to poor quality pages in hopes of getting out of the supplemental index, so a page change alone can’t be the sole criteria for moving a URL out of the supplemental index.

Promises aside, site owners should be able to know if they have page in the supplemental index. Ideally, I’d like to see a tool within Google Webmaster Central that would provide a list of these, or a percentage of pages from a site that are deemed supplemental as a health check, or both.

In talking with Matt, he expressed concern that this type of tool might cause site owners to needlessly fixate too much on supplementals as many have already been doing and similar to how many have obsessed for years over PageRank. Still, he said that Google would probably come up with a way for people to perform a query within Google Webmaster Central or some other method to find out if a page or pages are in the supplemental index.

Let’s hope that happens and soon. If the labels are going, an alternative method should be brought in as fast as possible. And despite concerns that site owners might obsess, I’d still like some nice reporting. After all, Google Webmaster Central already tells site owners their PageRank distribution. Reporting metrics on supplemental indexing makes as much sense.

Finally, remember that example above with the URL showing a supplemental label? Here it is for a regular search, for pagerank, appearing without the label:

Right now, the page ranks around 60 — pretty bad if you’re looking for traffic, but pretty good for a fairly new article on a very popular subject. In addition, doing some testing, I can tell that it is outranking some pages that are in the primary index. So being in the supplemental index clearly isn’t the kiss of death — but you’d still like to know the status. Providing that would help Google better prove its point. If people can easily tell that a supplemental page is outranking a page in the primary index, then concerns will actually ease. Just dumping the label alone won’t make those concerns go away.

Postscript: Google sent along this tip for finding supplemental pages now that the label is going, which I think will be part of what Matt will be posting on his own blog later.

First, get a list of all of your pages. Next, go to the webmaster console [Google Webmaster Central] and export a list of all of your links. Make sure that you get both external and internal links, and concatenate the files.

Now, compare your list of all your pages with your list of internal+external backlinks. If you know a page exists, but you don’t see that page in the list of site with backlinks, that deserves investigation. Pages with very few backlinks (either from other sites or internally) are also worth checking out.

I’d still rather go to to Google Webmaster Central, select the "export supplemental URLs" option and get a list that way!