Google Adds, Then Pulls, Advice Not To Block Ad Landing Pages From Its Crawler

googlebotOn Wednesday/Thursday of last week, Google added a new guideline to their Webmaster Guidelines to “not block a destination URL for a Google Ad product” via your robots.txt file. 24-hours later, Google reversed that guideline and put the webmaster guidelines back to exactly how they were Wednesday morning.

Here was the addition and then what they removed:

Make efforts to ensure that a robots.txt file does not block a destination URL for a Google Ad product. Adding such a block can disable or disadvantage the Ad.

We asked Google why did they make the change and then later, when we noticed the guideline was reversed, why did they reverse it?

We were considering something along these lines since advertising programs (not just ours) often need to crawl the landing pages, but we decided not to add it to the webmaster guidelines. Sorry for the confusion there.

As you may know, Google has told us they want us to make sure ads do not influence the search results and to block ads using the robots.txt file or use the nofollow. In fact, it is one of the existing guidelines:

Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, Google’s AdSense ads and DoubleClick links are blocked from being crawled by a robots.txt file.

But then they added this new guideline that communicated almost the exact opposite.

To be fair, Google’s ad ranking algorithm does need to detect landing page quality and thus needs to crawl. But as you can see, this can get confusing when placed in the guidelines.

As you can see, this guideline change and reversal is very interesting. It may have been a complete accident and we shouldn’t bother looking more at it, or it may be two different departments butting heads at the company.

Related Topics: Channel: SEO | Google: SEO | Google: Webmaster Central | Top News


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide