Google: News Sites May Have Seen A Decline In GoogleBot Crawling In March

Google’s Matt Cutts tweeted that news related sites, such as this one, may have noticed a drop in the crawl statistics within Google Webmaster Tools.

If you did notice that drop, Matt says you do not need to worry. Google News added if-modified-since support in March, which would be the reason for the crawl changes. Matt added that “Google News doing smarter crawling now” and thus being more efficient.

I’ve checked the crawl stats for seroundtable.com, a Google News publisher, and indeed, the crawl stats have declined significantly in March 2014:

To learn more about how GoogleBot handles the if-modified-since tag, see this blog post.

Here is Matt’s set of tweets:

If you have a news site and it looks like Google is crawling less since ~3/7, don’t worry. Google News added if-modified-since support then.

— Matt Cutts (@mattcutts) April 8, 2014

So it’s just Google News doing smarter crawling now. More info about if-modified-since: https://t.co/1TrtFVnypD

— Matt Cutts (@mattcutts) April 8, 2014