Twitter recently updated its robots.txt file and, though the change opens up millions of pages to being crawled, there's no guarantee that the main search engines want what Twitter is offering. The Sociable seems to have been the first to notice Twitter's robots.txt changes, which now specifically allow Google, Bing, Yahoo, Yandex and other bots to crawl through some of Twitter's search results pages. Twitter: Change Made To Help With Discovery Twitter confirmed the change to us, saying: This change will help people find popular and helpful Twitter pages, such as the #olympics hashtag pa [...]

Related Topics: Channel: SEO | Google: SEO | Google: Web Search | Microsoft: Bing | SEO: Blocking Spiders | Top News | Twitter | Twitter: Search

Technical SEO topics such as pagination are near and dear to my heart. This article will build upon and update my previous treatment of pagination and SEO. I've written and presented often on pagination for SEO. Why so much attention on this subject? The reason is simple: it can be a big, hairy deal for sites. It's right up there with faceted navigation as one of the most problematic crawling and indexing issues for large-scale SEO. It's a tactic (actually a set of tactics) that our teams are continually evolving, testing, and refining. So it was "double prizes" when Google announced [...]

Related Topics: Channel: SEO | Enterprise SEO | SEO - Search Engine Optimization | SEO: Blocking Spiders | SEO: Duplicate Content | SEO: Redirects & Moving Sites

As you know, there are many sites going black to protest SOPA and PIPA. Google has already offered blackout SEO advice but they decided to take it one step further by slowing down their spiders today. Pierre Far from Google posted on his Google+ that Google is slowing down GoogleBot's crawl activity to reduce the effect on their site's search rankings, if they did not follow the Google SEO advice from yesterday. Pierre Far said: Hello webmasters! We realize many webmasters are concerned about the medium-term effects of today's blackout. As a precaution, the crawl team at Google has conf [...]

Related Topics: Channel: SEO | Google: SEO | SEO: Blocking Spiders

A number of websites are (or were) planning to "go black" this week while the U.S. Congress discusses issues related to the Stop Online Piracy Act (SOPA) and the Protect IP Act (PIPA). The website blackouts are part of a larger social media effort against the bills that our Greg Finn wrote about this morning on Marketing Land. You may be thinking about joining the website blackout movement, but yikes ... what about the SEO implications? How do you take your site offline in protest without messing up your visibility in Google's search results? Well, Google's Pierre Far shared several tips [...]

Related Topics: Channel: SEO | Google: SEO | SEO: Blocking Spiders | SEO: Duplicate Content | Top News

This morning we reported that the comments on Facebook are being indexed by Google. Google's Matt Cutts just confirmed on Twitter that Google is now able to "execute AJAX/JS to index some dynamic comments." This gives Google's spider, GoogleBot, the ability to read comments in AJAX or JavaScript, such as Facebook comments or Disqus comments and others that are dynamically loaded via AJAX or JavaScript. In addition, this means, Google is better at seeing the content behind more of your JavaScript or AJAX. Postscript: Google now has an official blog post up with more details. Related [...]

Related Topics: Channel: SEO | Google: SEO | SEO: Blocking Spiders | SEO: Flash | Top News

This morning, James Breckenridge discovered a loophole within Google's Webmaster Tools that allowed anyone to remove any site from Google. Both James and I sent this information to Google as soon as we heard of it. After several hours, Google has told us, "we're still investigating this report, and to be cautious we disabled all URL removals earlier this morning." So now, if you even own a site, you won't be able to remove the site or pages from the site using Google's URL removal tool. How did this loophole work? Pretty simple as James described. You use the following URL when logged [...]

Related Topics: Channel: SEO | Google: SEO | Google: Webmaster Central | SEO: Blocking Spiders | SEO: Redirects & Moving Sites | SEO: Spamming | Top News

Google announced on the Webmaster blog that they have removed a requirement for removing URLs via Google Webmaster Tools. Google no longer requires you to block access to the URL you want to remove from Google prior to submitted the URL removal request. Google said they have eliminated "the requirement that the webpage's URL must first be blocked by a site owner before the page can be removed from Google's search results." Why did Google drop this requirement? Simply because since you have already validated and verified you are the owner of the site, Google felt it was redundant to requi [...]

Related Topics: Channel: SEO | Google: SEO | Google: Web Search | Google: Webmaster Central | SEO: Blocking Spiders

Last year, Google published an SEO Report Card of 100 Google properties. In it, they rated themselves on how well the sites were optimized for search. Google's Matt Cutts presented the results at SMX West 2010 in Ignite format. He noted that not every Googler is an expert in search and search engine optimization. Googlers who don't work in search don't get preferential treatment from those who do and just like any site on the internet, sometimes things aren't implemented correctly. Just because a site is owned by Google doesn't mean it's the best example of what to do in terms of SEO. This [...]

Related Topics: Channel: SEO | Features: Analysis | Google: SEO | How To: SEO | SEO: Blocking Spiders

The Google Webmaster Central blog announced Google has published a new set of documents in the Google Code section on how to control Google from crawling and indexing your site. You can read the set of documents over here. The technical documents are broken down into five section: Getting Started Robots.txt specification Robots meta tag and X-Robots-Tag specification Google's crawlers References I personally printed them out as my weekend reading. [...]

Related Topics: Channel: SEO | Google: SEO | SEO: Blocking Spiders

Malcolm Coles spotted that the Daily Mail, one of the UK's largest papers, changed their robots.txt file to include a line which reads: # August 12th, MailOnline are looking for a talented SEO Manager so if you found this then you're the kind of techie we need!# Send your CV to holly dot ward at mailonline dot co dot uk How clever! They suspect some of the best SEOs out there would be sniffing around their robots.txt file and used it to recruit a new SEO manger. If anything, it is getting the word out there via the press that they are looking for a new SEO. This reminds me of when Go [...]

Related Topics: Channel: SEO | SEM Industry: General | SEO: Blocking Spiders

That's one of the big questions people are asking after yesterday's launch of Facebook Questions. While many have assumed the answer would be "yes," a Facebook spokesperson tells us that assumption is wrong. Currently, search engines cannot access questions and answers through our Questions product. That may be something we consider for the future but have no current plans to allow it. Facebook is blocking search engines by only showing Questions to logged-in users. Sure enough, a site:facebook.com/questions/ search on Google shows only a handful of results, none of which are actuall [...]

Related Topics: Channel: SEO | Facebook | Search Engines: Answer Search Engines | Search Engines: Help Engines | SEO: Blocking Spiders | Top News | Yahoo: Answers

The home page of OpenOffice.org, the well-known Microsoft Office competitor, is missing from Microsoft's Bing search engine. While it sounds suspicious, the problem has nothing to do with Bing itself -- it's a technical problem on OpenOffice.org's end. Ian McAnerin noticed earlier today that OpenOffice.org doesn't show up in Bing on searches for [open office] and [openoffice.org]. He wonders if Bing is "allowing its results to be unduly influenced by either money or corporate policy." But, upon further digging with some help from SEL's Vanessa Fox, that's not the case. To be clear: Page [...]

Related Topics: Channel: SEO | Microsoft: Bing | SEO: Blocking Spiders

Google announced they have added a new supported user agent that can be used to block Google from indexing stories specifically on Google News. You can now specific the user agent Googlebot-News in your robots.txt file to exclude documents from being crawled and displayed in Google News. Google has several user agents for controlling how Google has access to your content. They have Googlebot, Googlebot-Image and so on. Each can be used to block specific content from showing up in specific Google search properties. In an effort to please some of the news providers, Google announced yes [...]

Related Topics: Channel: SEO | Google: News | Google: SEO | Google: Webmaster Central | SEO: Blocking Spiders | Top News

In the battle between search engines and some mainstream news publishers, ACAP has been lurking for several years. ACAP -- the Automated Content Access Protocol -- has constantly been positioned by some news executives as a cornerstone to reestablishing the control they feel has been lost over their content. However, the reality is that publishers have more control even without ACAP than is commonly believed by some. In addition, ACAP currently provides no "DRM" or licensing mechanisms over news content. But the system does offer some ideas well worth considering. Below, a look at how it [...]

Related Topics: Channel: SEO | Features: Analysis | Google: News | Legal: Crawling & Indexing | SEO: Blocking Spiders | Top News

The Robots Exclusion Protocol (REP) is not exactly a complicated protocol and its uses are fairly limited, and thus it’s usually given short shrift by SEOs. Yet there’s a lot more to it than you might think. Robots.txt has been with us for over 14 years, but how many of us knew that in addition to the disallow directive there’s a noindex directive that Googlebot obeys? That noindexed pages don’t end up in the index but disallowed pages do, and the latter can show up in the search results (albeit with less information since the spiders can’t see the page content)? That disallowed page [...]

Related Topics: All Things SEO | Channel: SEO | How To: SEO | SEO: Blocking Spiders

A month ago, Google, Yahoo and Microsoft announced they will be supporting a new canonical tag that allows you to tell search engines that page X is a duplicate page to page Z. In a way, it is a 301 redirect, without the physical redirect. The tag is incredibly powerful, as are 301 redirects and using this tag should be done with caution and slowly. Matt Cutts posted a new video explaining how one should go about using this tag, being that it is so new. Here is the video: [...]

Related Topics: Channel: SEO | Google: SEO | Google: Webmaster Central | SEO: Blocking Spiders | SEO: Duplicate Content | SEO: Redirects & Moving Sites | SEO: Submitting & Sitemaps | SEO: Tagging | SEO: Titles & Descriptions

The Live Search Blog announced they are letting a new robot loose. The new search engine crawler is named msnbot/2.0b and will be added to the army of current MSN spiders, currently named msnbot/1.1. The new spider is currently being tested but will ultimately replace the old spider. The new spider will respect the current robots.txt protocol set up for MSNBot, so no need to set up anything new in your robots.txt file. In addition, Microsoft promised to crawl slowly in their msnbot/2.0b tests. MSNBot/1.1 is not that old. It was added back in February of this year and introduced HTTP [...]

Related Topics: Channel: SEO | Microsoft: Bing | SEO: Blocking Spiders

I reported at the Search Engine Roundtable this morning that Google said if your robots.txt is unreachable, your site might not make it into the Google index. By unreachable, Google means that if your server simply times out and does not return any server response when Googlebot attempts to access your robots.txt file, then it might not include any of your pages in their index. Googler John Mueller explained that Google tends to lean on the "safe" side when this situation pops up. When I showed this to Danny, he felt it was ironic that if Google can't read what you want to block, it might bl [...]

Related Topics: Channel: SEO | Google: SEO | SEO: Blocking Spiders

Last week, the three major search engines came together to say how they agree -- and disagree -- over the Robots Exclusion Protocol. It's such an important standard, one every webmaster should understand. To help, Vanessa Fox has compiled an extensive and outstanding overview of it at Jane & Robot in her Managing Robot's Access To Your Website post. The tutorial takes you through key areas such as: A nice chart showing what you can block using either robots.txt or the meta robots tag for each major search engine. It also covers other things like reverse DNS lookup to verify a crawler's [...]

Related Topics: Channel: SEO | SEO: Blocking Spiders

Today, Google, Yahoo!, and Microsoft have come together to post details of how each of them support robots.txt and the robots meta tag. While their posts use terms like "collaboration" and "working together," they haven't joined together to implement a new standard (as they did with sitemaps.org). Rather, they are simply making a joint stand in messaging that robots.txt is the standard way of blocking search engine robot access to web sites. They have identified a core set of robots.txt and robots meta tag directives that all three engines support: Google and Yahoo! already supported and doc [...]

Related Topics: Channel: SEO | SEO: Blocking Spiders