Pierre Far from Google posted on his Google+ that Google is slowing down GoogleBot’s crawl activity to reduce the effect on their site’s search rankings, if they did not follow the Google SEO advice from yesterday.

Pierre Far said:

Hello webmasters! We realize many webmasters are concerned about the medium-term effects of today’s blackout. As a precaution, the crawl team at Google has configured Googlebot to crawl at a much lower rate for today only so that the Google results of websites participating in the blackout are less likely to be affected.

Related Articles

• What All Marketers Need To Know About SOPA – The Stop Online Piracy Act
• #BlackoutSOPA: A Look At The Social Media Movement That Helped Stall The SOPA Legislation
• How To Blackout Your Site (For SOPA/PIPA) Without Hurting SEO
• Google Blackens Its Logo To Protest SOPA/PIPA, While Bing & Yahoo Carry On As Usual
• Why The Web Is Going Dark Over SOPA & PIPA

You may be thinking about joining the website blackout movement, but yikes … what about the SEO implications? How do you take your site offline in protest without messing up your visibility in Google’s search results?

Well, Google’s Pierre Far shared several tips earlier today on Google+ in a post called “Website outages and blackouts the right way.”

In short, the advice is to use a 503 HTTP status code to tell spiders that the website is temporary unavilable. With a 503 status, Google won’t index the content (or lack thereof if you’re blacking out your site) and it won’t consider the site as having duplicate content issues (when all of the pages are blacked out).

But Far adds a couple important caveats to this advice regarding the robots.txt file and what will happen in Webmaster Tools if Google finds your site blacked out. Another Googler, John Mueller, adds additional information in the comments, so you’ll want to read the original Google+ post if you’re thinking about blacking out your website this week for SOPA, or in the future for any other reason.

Of course, also keep in mind that Bing may not handle things the same way if you do blackout your site.

This gives Google’s spider, GoogleBot, the ability to read comments in AJAX or JavaScript, such as Facebook comments or Disqus comments and others that are dynamically loaded via AJAX or JavaScript. In addition, this means, Google is better at seeing the content behind more of your JavaScript or AJAX.

Postscript: Google now has an official blog post up with more details.

Related Stories:

• Google Offers A Proposal To Make AJAX Crawlable
• Google Offers SEO Advice On AJAX Coding
• Google May Be Crawling AJAX Now – How To Best Take Advantage Of It
• It’s Official: Google’s Proposal For Crawling AJAX URLs is Live
• An Update On Javascript Menus And SEO
• Google Now Crawling And Indexing Flash Content

Both James and I sent this information to Google as soon as we heard of it. After several hours, Google has told us, “we’re still investigating this report, and to be cautious we disabled all URL removals earlier this morning.” So now, if you even own a site, you won’t be able to remove the site or pages from the site using Google’s URL removal tool.

How did this loophole work? Pretty simple as James described. You use the following URL when logged into Google Webmaster Tools:

https://www.google.com/webmasters/tools/removals-request?hl=en&siteUrl=http://{YOUR_URL}/&urlt={URL_TO_BLOCK}

Then replace {YOUR_URL} with a URL you control within Webmaster Tools, and replace {URL_TO_BLOCK} with the URL of the site you want to block.

You could block a whole site, section or single page this way, based on how you entered the URL. To block a site, use the top level domain (E.g. http://www.someurl.com/), to block a section (subfolder) use a subfolder URL (E.g. http://www.someurl.com/somefolder/) and to block a page use the specific page URL (E.g. http://www.someurl.com/somefolder/somepage.html).

I am waiting an update from Google on why this happened, if site’s were impacted and how long this was an issue.

Postscript:: Google sent us a statement that they have fixed the issue. A Google spokesperson said:

We’ve confirmed that there was an issue within the URL removal feature in our Webmaster Tools and have already pushed out a fix and re-enabled URL removals.

The URL removal feature keeps detailed records, so we’re currently reprocessing earlier removal requests to ensure their validity. Our initial examination has shown only a limited impact.

Google no longer requires you to block access to the URL you want to remove from Google prior to submitted the URL removal request. Google said they have eliminated “the requirement that the webpage’s URL must first be blocked by a site owner before the page can be removed from Google’s search results.” Why did Google drop this requirement? Simply because since you have already validated and verified you are the owner of the site, Google felt it was redundant to require you to block the URL to prove again you are the site owner. Google explained:

You’ve already verified ownership of the site, we can eliminate this requirement to make it easier for you, as the site owner, to remove unwanted pages (e.g. pages accidentally made public) from Google’s search results.

Please note, these URL removals are only temporary and last 90-days. If you want the URLs and pages to never show in Google, you need to permanently remove them by 404ing the page or blocking them via robots.txt file or a noindex meta tag.

Related Stories:

• Removing Pages From Google: A Comprehensive Guide For Content Owners
• Google Releases Improved Content Removal Tools
• Google Lets You Tell Them Which URL Parameters To Ignore
• Up Close & Personal With Robots.txt
• Removing Your Personal Information From Google

This morning Rishi Lakhani tweeted about Google Translate pages appearing in Google search results. As you can see in the example below, pages with individual translation requests have been indexed.

All of the URLs that include a parameter seem to be individual translations. For instance, http://translate.google.com/?q=ART# displays as follows:

The problems with these types of pages being indexed in search engines is twofold:

• The Google webmaster guidelines say that Google doesn’t want to show search results in its search results and recommends that content owners block search results on their site from being indexed using robots.txt or a meta robots tag.
• That same guideline recommends blocking autogenerated pages from being indexed and a Google Webmaster Central blog a few months ago provided recommendations for handling machine-translated text so that it didn’t appear in search results.

A site owner might also want to block these types of pages from being crawled and indexed to increase crawl efficiency and ensure the most valuable pages on the site are being crawled and indexed instead.

I asked Google about this and they confirmed that indeed it was simply a matter of the Google Translate team not being aware of the issue and said they would resolve it.

Blocking Autogenerated Search Pages From Being Indexed

In the case of Google Translate, the ideal scenario is that the main page and any secondary pages (such as this tools page) be indexed, but that any pages from translation requests not be indexed.

Using robots.txt

The best way to do this would be to add a disallow line in the robots.txt file for the site that blocks indexing based on a pattern match of the URL query parameter. For instance:

Disallow: /*q=

This pattern would prevent search engines from indexing any URLs containing q=. (The * before the q= means that the q= can appear anywhere in the URL.)

In the case of translate.google.com (and all related TLDs), the robots.txt file that exists for the subdomains seems to be copied from www.google.com. Remember that search engines obey the robots.txt file for each subomain separately. Using the same robots.txt file for a subdomain that’s used for the www variation of the domain could have unintended consequences because the subomain likely has an entirely different folder and URL structure. (You can always check the behavior of your robots.txt file using Google Webmaster Tools.)

Adding the disallow pattern shown above to the www.google.com/robots.txt file would not work as search engines wouldn’t check that file when crawling the translate subdomain and in would instead cause search engines not to index URLs that match the pattern on www.google.com.

translate.google.com (and all google.com subdomains should have their own robots.txt file that’s customized for that subdomain.

Using the meta robots tag

If Google isn’t able to create a separate robots.txt file for the translate subdomain, they should first remove the file that’s there (and from other subdomains as well, as it could be causing unexpected indexing results for those subdomains). Then, they should use the meta robots tag on the individual pages they want blocked. Since the pages in question are dynamically generated, the way to do this would be to add logic to the code that generates these pages that writes the robots meta tag to the page as its created. This tag belongs in the <head> section of the page and looks as follows:

<meta="robots" content="noindex">

The technical documents are broken down into five section:

• Getting Started
• Robots.txt specification
• Robots meta tag and X-Robots-Tag specification
• Google’s crawlers
• References

I personally printed them out as my weekend reading.

# August 12th, MailOnline are looking for a talented SEO Manager so if you found this then you’re the kind of techie we need!
# Send your CV to holly dot ward at mailonline dot co dot uk

How clever! They suspect some of the best SEOs out there would be sniffing around their robots.txt file and used it to recruit a new SEO manger. If anything, it is getting the word out there via the press that they are looking for a new SEO.

This reminds me of when Google added the User-agent: zombies Disallow: /brains to their robots.txt file on Halloween. Also, in 2006, Brett Tabke used his robots.txt file as his blog for a period of time.

I do not believe anyone every used it to hire an SEO before.

That’s one of the big questions people are asking after yesterday’s launch of Facebook Questions. While many have assumed the answer would be “yes,” a Facebook spokesperson tells us that assumption is wrong.

Currently, search engines cannot access questions and answers through our Questions product. That may be something we consider for the future but have no current plans to allow it.

Facebook is blocking search engines by only showing Questions to logged-in users. Sure enough, a site:facebook.com/questions/ search on Google shows only a handful of results, none of which are actually Q&A from the first 24 hours that the beta has been open.

The same search produces zero results on both Yahoo and Bing.

Facebook’s move is unusual. As far back as 2007, the company was starting to open up some content to search engines — a trend that’s continued more recently with various agreements to let search engines access certain user content. For years, Yahoo Answers has been a pretty formidable rankings powerhouse, and no doubt gets a substantial amount of traffic via search engines.

It’s odd that Facebook would want to ignore that traffic source entirely. Don’t be surprised if this policy changes.

Postscript, July 30: Experian Hitwise has responded to my claim near the end of this article about how much traffic Yahoo Answers gets via search engines. In a tweet this morning, Hitwise reports that “62% of upstream visits to Yahoo Answers came via Google last week.”

Ian McAnerin noticed earlier today that OpenOffice.org doesn’t show up in Bing on searches for [open office] and [openoffice.org]. He wonders if Bing is “allowing its results to be unduly influenced by either money or corporate policy.” But, upon further digging with some help from SEL’s Vanessa Fox, that’s not the case.

To be clear: Pages from the openoffice.org domain do show up in Bing — a [site:openoffice.org] search proves that. But the home page itself is nowhere to be found.

It seems as though the problem is simply due to a technical misconfiguration on the openoffice.org servers. This issue is impacting Yahoo’s index as well as Bing’s. When you navigate to openoffice.org as a user, you see the home page as you should. If you change the user agent to Googlebot (Vanessa used the User Agent Switcher Firefox plugin), you see the same nicely rendered home page.

However, if you change the user agent to either MSNbot or Yahoo Slurp, you see a 403 access denied error.

You can see this more clearly in the HTTP response from the server (using a tool such as the Live HTTP Headers Firefox plugin). Accessing the page as Googlebot returns the following (shortened for space; note the 304 rather than 200 response simply because Vanessa had visited the page before as Googlebot):

Host: www.openoffice.org

GET / HTTP/1.1
Host: openoffice.org

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

If-Modified-Since: Mon, 29 Mar 2010 12:58:51 GMT

HTTP/1.0 304 Not Modified

Whereas accessing the page as MSNbot looks like this:

http://www.openoffice.org/

GET / HTTP/1.1

Host: openoffice.org

User-Agent: msnbot/1.1 (+http://search.msn.com/msnbot.htm)

HTTP/1.0 403 Forbidden

How did the server get set up this way? Any number of explanations are possible. Sometimes this happens when the host notices overactive crawling from particular bots and blocks them. This is always something that a site owner who uses shared hosting should watch out for (as the result is that your site gets dropped from that search engine’s index). In this case, Open Office likely manages their own servers, but they may not be blocking Microsoft and Yahoo purposely. A piece of server software could have easily been misconfigured accidentally.

A Microsoft spokesperson tells us: “We’re reaching out to them now to try and resolve the issue.”