SEO > SEO: Blocking Spiders
Mar. 27, 2008 at 5:39pm by Danny Sullivan
Google Offers Robots.txt Generator
Google's rolled out a new tool at Google Webmaster Central, a robots.txt generator. It's designed to allow site owners to easily create a robots.txt file, one of the two main ways (along with the meta robots tag) to prevent search engines from indexing content. Robots.txt generators aren't new. You can...
See Related Stories In: Google: SEO, Google: Webmaster Central, SEO: Blocking Spiders
Feb. 25, 2008 at 8:53am by Barry Schwartz
SEOs Want The NOINDEX Tag To Not Show A Page In The Index
Matt Cutts of Google posted a blog entry asking SEOs how they want Google to handle the NOINDEX meta tag. If you use the NOINDEX meta tag now, Google won't show the page in any way in the Google index -- not even a "link only" listing. Matt asks SEOs...
See Related Stories In: Google: SEO, SEO: Blocking Spiders
Dec. 5, 2007 at 2:08pm by Barry Schwartz
Yahoo Search Weather Update & Support For X-Robots Tag
The Yahoo Blog issued a weather report for changes to rankings in Yahoo Search, along with news that they are now supporting the X-Robots-Tag directive -- a way to control indexing of content that cannot accept meta robots tags....
See Related Stories In: SEO: Blocking Spiders, Yahoo: Search
Nov. 29, 2007 at 12:02pm by Danny Sullivan
ACAP Launches, Robots.txt 2.0 For Blocking Search Engines?
After a year of discussions, ACAP -- Automated Content Access Protocol -- was released today as a sort of robots.txt 2.0 system for telling search engines what they can or can't include in their listings. However, none of the major search engines support ACAP, and its future remains firmly one...
See Related Stories In: SEO: Blocking Spiders
Nov. 15, 2007 at 1:19pm by Barry Schwartz
Robots.txt Study Shows Webmasters Favor Google; BotSeer Robots.txt Search Engine Released
The Pennsylvania State University conducted a study that showed webmasters favored Google over other search engines in terms of allowing access to their web sites. An associated BotSeer search engine that allows searching across a collection of robots.txt files was also released....
See Related Stories In: SEO: Blocking Spiders, Search Engines: Other Search Engines, Stats: Popularity
Aug. 16, 2007 at 1:01pm by Barry Schwartz
How Proxy Hacking Can Hurt Your Rankings & What To Do About It
Google Proxy Hacking: How A Third Party Can Remove Your Site From Google SERPs by Dan Thies gives us a detailed look at the serious dangers of proxy hacking. Dan's detailed article shows the history on how he discovered the issue. He then goes into why the hacking currently works...
See Related Stories In: Google: SEO, SEO: Blocking Spiders, SEO: Redirects & Moving Sites, SEO: Spamming
Aug. 16, 2007 at 8:58am by Barry Schwartz
Google Enhances Webmaster Central's Robots.txt Analysis Tool
The Google Webmaster Central Blog announced improvements they have made to the robots.txt analysis tool. The tool now recognizes all sitemap declarations and relative URLs. So now the tool will report the validity of all sitemaps URLs plus show data for relative URLs. In addition, Google has expanded the reporting...
See Related Stories In: Google: SEO, Google: Webmaster Central, SEO: Blocking Spiders
Jul. 27, 2007 at 2:52pm by Barry Schwartz
Google's "Unavailable After" META Tag Now Live
Google's Dan Crow announced today that the unavailable_after META tag is now live and operational. Google To Add "Unavailable After" META Tag from about two weeks ago, explains in detail more about this tag and how it can be used....
See Related Stories In: Google: SEO, SEO: Blocking Spiders, SEO: Titles & Descriptions
Jul. 17, 2007 at 11:15am by Barry Schwartz
More Info On Google's Unavailable After Meta Tag & New X-Robots-Tag In Header Support
Last week we reported that Google was to add an "Unavailable After" META Tag. Since then, we've spoke to Dan Crow of Google, who provided more information on how to use it, as well information on a new way to send robots blocking info within HTTP headers....
See Related Stories In: Google: SEO, SEO: Blocking Spiders
Jul. 12, 2007 at 9:30am by Barry Schwartz
Google To Add "Unavailable After" META Tag
Getting Into Google by Jill Whalen reports Dan Crow, director of crawl systems at Google, saying that Google is releasing a new META tag named "unavailable_after." The "unavailable_after" tag will allow you to tell Google when Googlebot should no longer crawl that page. Jill explains that this tag comes in...
See Related Stories In: Google: SEO, SEO: Blocking Spiders, SEO: Titles & Descriptions
May. 23, 2007 at 3:25am by Elliance
Search Illustrated: Blocking Search Engines With Robots.txt
While most of the time we want search engine crawlers to grab and index as much content from our web sites as possible, there are situations where we want to prevent crawlers from accessing certain pages or parts of a web site. For example, you don't want crawlers poking...
See Related Stories In: SEO: Blocking Spiders, Search Illustrated
May. 3, 2007 at 8:24am by Danny Sullivan
Belgian Papers Back In Google; Begin Using Standards For Blocking
Belgian newspapers that sued Google to be removed from its index are now back in, having agreed to use the commonly-accepted blocking standards that they initially rejected as not being legal. Google and the group representing the papers, Copiepresse, have issued a joint statement. That's below, along with a look...
See Related Stories In: Google: Business Issues, Google: Legal, Google: News, Legal: Copyright, Legal: Crawling & Indexing, SEO: Blocking Spiders
May. 2, 2007 at 1:23pm by Danny Sullivan
Yahoo Supports New Robots-Nocontent Tag To Block Indexing Within A Page
For over a decade, search engines have supported standards allowing you to prevent pages from being spidered or included within a search index. Today, Yahoo now supports a new twist -- a way to flag that part of your page shouldn't be included in an index. It's called the...
See Related Stories In: SEO: Blocking Spiders, Yahoo: SEO
Apr. 30, 2007 at 9:47pm by Danny Sullivan
From The Isn't It Ironic Dept: Google Product Search's Results Show Up In Google
Remember how Google said recently that it might crack down on listings pages that are simply search results themselves? Reader Michael Nguyen dropped an email today to point out how, ironically, Google is now listing pages from its own Google Product Search service exactly as it has warned others not...
See Related Stories In: Google: OneBox, Plus Box & Direct Answers, Google: Product Search, Google: SEO, SEO: Blocking Spiders, SEO: Spamming
Apr. 30, 2007 at 7:15am by Barry Schwartz
How Search Engines Handle The Nofollow Attribute
Loren Baker at Search Engine Journal has a nice write up on how the search engines handle the nofollow attribute now just over two years since it was introduced. Ask.com still does not follow the tag, so here are the takeaway for Google and Yahoo: Google won't follow the link,...
See Related Stories In: SEO: Blocking Spiders
Apr. 17, 2007 at 9:38pm by Danny Sullivan
Google Releases Improved Content Removal Tools
Google has rolled out new tools to help people quickly get content removed from its search engine. Those targeted at site owners allow for speedy removal of pages and cached copies of pages. Other tools allow those to request the removal of images or links to pages with personal information...
See Related Stories In: Google: SEO, Google: Webmaster Central, Legal: Privacy, SEO: Blocking Spiders
Apr. 16, 2007 at 1:15pm by Christine Churchill
Up Close & Personal With Robots.txt
The Robots.txt Summit at Search Engine Strategies New York 2007 was the latest in a series of special sessions with the intent to open a dialog between search engines representatives and web site publishers. Past summits featured discussion on comment spam on blogs, indexing issues and redirects. The subject of...
See Related Stories In: SEM Industry: Conferences, SEO: Blocking Spiders, SEO: Submitting & Sitemaps
Mar. 12, 2007 at 10:42am by Danny Sullivan
Google Warning Against Letting Your Search Results Get Indexed
The days of doing a Google search that brings up results leading to search results from other sites are heading for a close. Matt Cutts, in his Search Results In Search Results post today, points out a change to Google's guidelines that shows a crackdown on this type of material...
See Related Stories In: Google: SEO, SEO: Blocking Spiders, SEO: Spamming
Mar. 5, 2007 at 8:48pm by Danny Sullivan
Meta Robots Tag 101: Blocking Spiders, Cached Pages & More
Last week, I covered a new command for the meta robots tag -- one to prevent search engines from using Yahoo titles and descriptions. In doing that, a number of questions came up about the meta robots tag syntax itself. Google Webmaster Central has now posted "Using the robots meta...
See Related Stories In: Ask: SEO, Google: SEO, Google: Webmaster Central, Microsoft: Live Search SEO, SEO: Blocking Spiders, SEO: Titles & Descriptions, Yahoo: SEO
Feb. 28, 2007 at 2:06pm by Danny Sullivan
Yahoo Provides NOYDIR Opt-Out Of Yahoo Directory Titles & Descriptions
Yahoo! Search Support for 'NOYDIR' Meta Tags and Weather Update from the Yahoo Search Blog covers how at long last, you can now tell Yahoo to not use Yahoo Directory information to make a title and/or description for your web page listings. It also cover how Yahoo's currently doing a...
See Related Stories In: Google: SEO, Microsoft: Live Search SEO, SEO: Blocking Spiders, SEO: Titles & Descriptions, Yahoo: SEO
Feb. 27, 2007 at 3:47pm by Danny Sullivan
Squeezing The Search Loaf: Finding Search Engine Freshness & Crawl Dates
A reader emailed me today noticing that Google was showing a date next to his listing, which made me think this was a good time to revisit how, when and where search engines show crawl dates for pages. These dates are a useful way for site owners to understand how...
