Is Google Broken? Sites Big & Small Seeing Indexing Problems

No one seems to be immune from a Google indexing problem that has many site owners baffled. Blogs and websites, big and small, aren’t being indexed as quickly as they normally are — if they’re being indexed at all.

CNN.com, with its PageRank score of 10, is typically crawled frequently and deeply by Googlebot. But the most recent article added to its home page (under the “Latest news” title) still hasn’t been indexed by Google more than three hours later.

Here on Search Engine Land, the most recent article we’ve published was my piece about the new Chrome extension for reporting spam. That was published an hour ago, and at this moment, Google is only showing a FriendFeed post that links to our article.

That FriendFeed post seems to come from an RSS-based mashup that grabs posts from a variety of tech-related sites.

Perhaps related is this situation with Mashable’s post today about the Wikileaks Iraq document release; rather than indexing the actual article URL, Google has indexed the URL with several UTM tracking parameters for Google Analytics. The question mark in the green display URL (below) gives it away.

When Google Caffeine launched this past summer, faster indexing and a fresher index was Google’s big selling point. As Google’s Matt Cutts explained at the time, Caffeine meant “that all content … can be searchable within seconds after its crawled.”

Except not this week. And the indexing problems are affecting web site owners of all sizes. As Search Engine Roundtable reported this morning, there are active threads in the Google Webmaster Central forums about this, along with various blog posts out there, such as here and here. The reports there appear to be mainly from bloggers using Google’s blogspot.com platform, but as you can see above, the indexing trouble isn’t limited just to Blogger users.

In one of those threads, Google employee John Mueller says Google is following the user reports and “looking at ways to resolve the issues brought up here.” We’ll keep an eye on any new developments or updates.

If you’re seeing or experiencing indexing problems in Google, let us know in the comments.

Postscript by Barry Schwartz: John from Google replied to the thread in the Webmaster forums saying:

I just wanted to provide a short update. This issue should be resolved soon, but it will take a several days for everything to catch up and for these changes to be visible. I’ll provide another update after the weekend.

In the meantime, there’s no need to make any changes on your side. Content from your sites will continue to be indexed during this time (though perhaps at the moment not as quickly as before in some cases).

I’ve been in contact with the indexing and diagnostics teams since your initial reports — we are taking your reports seriously and doing what is necessary to resolve this problem as soon as possible. We love sending users to your sites quickly after you have published something too :-).

Just to be clear, the issues from this thread, which I have reviewed in detail, are not due to changes in our policies or changes in our algorithms; they is due to a technical issue on our side that will be visibly resolved as soon as possible (it may take up to a few days to be visible for all sites though). You do not need to change anything on your side and we will continue to crawl and index your content (perhaps not as quickly at the moment, but we hope that will be resolved for all sites soon). I would not recommend changing anything significantly at this moment (unless you spot obvious problems on your side), as these may result in other issues once this problem is resolved on our side.

Postscript #2 by Barry: John from Google commented on the Search Engine Roundtable claiming it has been fixed. I did spot check the forum thread and many webmasters and bloggers confirmed. John did add:

If you are still seeing issues with indexing of new content, it’s likely to be a different problem & I would recommend posting in our help forum with all the details.