Topic: Web crawler

A Big Roundup Of Link Building Tools

Ask any link builder what link issues they’re consistently asked about and one of them will be linking tools. Link building is such a time consuming and detailed process that it’s only natural people look for tools and techniques to maximize their time. Those who spend a lot of time link building are familiar with […]

MSNbot 1.1: Live Search Implements A More Efficient Crawl

Today, Microsoft announced changes to their Live Search crawler intended to reduce bandwidth resources during the crawl of a site. MSNbot (upgraded to version 1.1) now supports both HTTP compression and conditional get. The post on the Live Search Webmaster Center blog describes each feature in detail and includes links to tools you can use […]

Yahoo Search Update: First Update Of 2008

Yahoo has announced the first Yahoo Search update of 2008. Sharad Verma said Yahoo has been rolling out updates to their “crawling, indexing and ranking algorithms” over the past couple days. Yahoo tells us the update will be done soon, but you may notice “some ranking changes and page shuffling in the index.” I reported […]

Microsoft To Buy Fast Search For $1.23 Billion

Microsoft Agrees to Buy Fast Search for $1.23 Billion from Bloomberg reports that Microsoft is buying FAST Search. The Norwegian search company, Fast Search & Transfer ASA, will be bought by Microsoft for 6.6 billion kroner, or $1.23 billion. Microsoft will buy the company via the Oslo Stock Exchange, share by share. Here is the […]

Google Improves Results For Supplemental Pages

According to a new post on the Google Webmaster Central blog, the supplemental index is no longer, well, supplemental. Google has long had a two-tiered index and webmasters have generally feared the second, supplemental tier. A Forbes article earlier this year called it “Google Hell“, as historically, those pages weren’t crawled as often as those […]

Sub Domains To Be Treated As Folders By Google

Matt Cutts of Google said at PubCon that Google will be treating sub domains similar to how they treat folders on a site. I reported this at the Search Engine Roundtable on a post by Tedster at WebmasterWorld, where Tedster said: News flash from Las Vegas PubCon. Matt Cutts informed us that Google will very […]

SEO

ACAP Launches, Robots.txt 2.0 For Blocking Search Engines?

After a year of discussions, ACAP — Automated Content Access Protocol — was released today as a sort of robots.txt 2.0 system for telling search engines what they can or can’t include in their listings. However, none of the major search engines support ACAP, and its future remains firmly one of "watch and see." Below, […]

SEO

Yahoo Search Updates Algorithm, Index, & Crawl

The Yahoo Search Blog has posted that they are currently undergoing an index, crawl, and algorithm update. Priyank Garg of Yahoo said: Over the last few days, we’ve been rolling out some changes to our crawling, indexing and ranking algorithms. While we expect the update will be completed soon, as you know, throughout this process […]

Google Adds Sitelinks Control To Webmaster Tools & Much More

Google has informed us that today they are releasing a new update to Google Webmaster Central’s tools. The new features include a way to manage your Google Sitelinks, more historical query statistics, an extension of sitemaps to support code search, an improved message center and a warning that may help notify you of possible duplicate […]

PPC

Big SEO: Automate Or Die Trying

If you are one of the growing numbers of in-house SEOs working on larger web sites, things can get a bit “scratch-your-head, where-do-I-start” difficult. If you’re managing tens of millions of pages, or even tens of thousands of pages, there’s no doubt you need to automate things. Even those working on smaller sites that produce […]

SEO

Yahoo Search Updating Index & Algorithm

We have been noticing reports from several sources of a Yahoo Search index and algorithm update. It is important to note that Yahoo has not yet posted their “weather report” at the Yahoo Search Blog yet. However Aaron Wall, Loren Baker and several forums have noticed major changes at Yahoo Search.

SEO

Yahoo Slurp Gets More Efficient & New Index Update

The Yahoo Search Blog announced that they have made major changes to their crawling process, plus rolled out a new update to their crawling, indexing and ranking systems. Recently, there have been many complaints that Yahoo’s crawler, Yahoo Slurp, is crawling sites too heavily. Yahoo says it has taken this feedback and has improved their […]

Google Enhances Webmaster Central’s Robots.txt Analysis Tool

The Google Webmaster Central Blog announced improvements they have made to the robots.txt analysis tool. The tool now recognizes all sitemap declarations and relative URLs. So now the tool will report the validity of all sitemaps URLs plus show data for relative URLs. In addition, Google has expanded the reporting to include data for not […]

Supplemental Results and Google’s Extended Databases

Until very recently, you might have seen a label next to a search result in Google that indicated it was a “supplemental” result. A couple of patents from Google, one of which was granted this week and one from earlier this year, discuss how a search query might return results from an extended database that […]

Google’s Last Visited Time Stamp Gets Down To The Minute

Google Cache Showing Last Retrieve Dates in Minutes at Search Engine Roundtable shows how Google is now showing when it last visited some pages on a per minute or per hour basis, rather than in the traditional per day style. For example, this is how Google has normally shown last visit dates for some pages: […]

SEO

Yahoo July 2007 Search Update

The Yahoo Search Blog announced that they are currently rolling out a web search update. Priyank Garg’s “weather report” said: We’ve been rolling out some changes to our fresh web data and crawling, indexing and ranking algorithms over the last few days. We expect the update will be completed by the weekend. You may see […]

Understanding Search Engines Duplicate Content Issues

I admit it. I am a search engine geek. Because I have a passion for understanding search usability, one of my particular interests is duplicate content filtering. If you want to really irritate searchers, present the same content to them in all or most of the top 10 positions in search results. In the past, […]

Google AdSense Launches Site Authentication Feature

The Inside AdSense blog announced that they have launched a new feature named “Site Authentication.” The Site Authentication feature enables you to give the AdSense crawler access to your password protected pages. All you need to do is provide AdSense with a username and password. Then the AdSense crawler will gain access to those protected […]

Google To Add “Unavailable After” META Tag

Getting Into Google by Jill Whalen reports Dan Crow, director of crawl systems at Google, saying that Google is releasing a new META tag named “unavailable_after.” The “unavailable_after” tag will allow you to tell Google when Googlebot should no longer crawl that page. Jill explains that this tag comes in handy when you have a […]

SEO

Yahoo Finalizes Slurps Move To New Address

The Yahoo Search Blog announced the transition of Yahoo Slurp, Yahoo’s web crawler, from inktomisearch.com to crawl.yahoo.net is now complete. The IP addresses and the user-agent are remaining the same but the DNS lookup is different, it is now crawl.yahoo.net. This does not apply to other Yahoo crawlers, including Yahoo China, Yahoo Shopping, Yahoo Travel […]

Google Checkout’s Title Goes Missing On Google

  Something look odd in the search results for google checkout? Yes, that’s the official Google Checkout home page listed first, but it’s most definitely NOT the right title for the Google Checkout page. That’s got folks at Digg wondering if Google Checkout has been hijacked, sparked from Jon West’s original post here. No. It’s […]

SEO

How Search Engines Handle The Nofollow Attribute

Loren Baker at Search Engine Journal has a nice write up on how the search engines handle the nofollow attribute now just over two years since it was introduced. Ask.com still does not follow the tag, so here are the takeaway for Google and Yahoo: Google won’t follow the link, Yahoo will (note) Google and […]

Google Releases Improved Content Removal Tools

Google has rolled out new tools to help people quickly get content removed from its search engine. Those targeted at site owners allow for speedy removal of pages and cached copies of pages. Other tools allow those to request the removal of images or links to pages with personal information about themselves, in the right […]

Search Engines Unite On Sitemaps Autodiscovery

Last November, Google, Microsoft and Yahoo united to support sitemaps, a standardized method of submitting web pages through feeds to the search engines. Today, the three are now joined by Ask.com in supporting the system and an extension of it called autodiscovery. This is where the major search engines will automatically locate your sitemaps file […]

PPC

Yahoo Site Explorer Takes Mobile Feeds, Gains Report Spam Button

Site Explorer Matures a Bit More and Accepts Mobile Feeds from the Yahoo Search Blog covers how the Yahoo Site Explorer tool now accepts mobile submissions and feeds, gains the ability to input URLs for deletion and remove more than five at a time (see our Yahoo Site Explorer Adds Features: Delete URL From Index […]

ZoomInfo Offers Free Company Profiles

ZoomInfo today launched its Business Information Search Engine, a service that offers information on more than 3.5 million companies. Although the company profiles are similar to those offered by Hoovers.com and other subscription-based providers, ZoomInfo business profiles are free. The profiles also differ in how they’re compiled. While most business information providers laboriously compile information […]

SEO

Yahoo Slurp Moves To crawl.yahoo.net

The Yahoo Search Blog announced that Yahoo Slurp, Yahoo’s web crawler, is moving from inktomisearch.com to crawl.yahoo.net. In short, the user agent for Yahoo’s crawler will remain “Yahoo! Slurp.” However, reverse DNS lookups on the IP address by the true Yahoo crawler will now resolve to crawl.yahoo.net. The transition should take “a few weeks.”

SEO: Real Skills That Can Protect Your Traffic

Maybe SEO isn’t rocket science, but the wrong SEO decision (say knowing the difference between a 301 and a 302 redirect) could be costly. That’s illustrated in a Wall Street Journal article today on news search engine Topix and SEO issues in general. A look at that, revisiting Google’s webmaster support efforts plus new confirmation […]

Google Warning Against Letting Your Search Results Get Indexed

The days of doing a Google search that brings up results leading to search results from other sites are heading for a close. Matt Cutts, in his Search Results In Search Results post today, points out a change to Google’s guidelines that shows a crackdown on this type of material may begin. More about what I’m talking about below, […]