Topic: Web crawler

Do Links From Expired Domains Count With Google?

From time-to-time, issues about how domain names can impact Google rankings come up. Is it true that if you buy a name, all the “link equity” that name has gained is lost? Below, a look at this and some related questions. Several years ago, Google sparked some concerns when it said that buying an established […]

SEO

Yahoo Announces March 2009 Search Update

A little over a week ago we reported on the March ’09 Yahoo Search update. Yahoo has finally confirmed the update with a weather report on the Yahoo Search Blog. Dan Rampton from Yahoo Search said: We’ll be rolling out some changes to our crawling, indexing and ranking algorithms over the next few days and […]

SEO

Yahoo! BOSS Expands Feature Set; Adds Usage Fees

Today, Yahoo! has announced several new features to their Build Your Own Search Service (BOSS) platform as well as usage fees. When BOSS launched in July 2008, the stated goal was to spur innovation and disrupt the search market. Will the additional of usage fees hinder that goal and become a barrier to entry for […]

Google Creates Smart Sitemap Generator Tool

The Google Webmaster Central blog announced that they have just released a new Sitemap generator that works on both Linux/Apache and Microsoft IIS Windows-based servers. The Sitemap generator is different and possibly smarter than other Sitemap tools because it uses web traffic, log files and other methods to find new or modified URLs. You can […]

SEO

SEO & Page Segmentation Analysis

If you’re looking for some fairly heavy SEO reading, something beyond the “Top 10-this” and “5-Tips-for-that” type of post, spend some time reading Dave Harry’s post on how search engines may get more granular with page segmentation analysis. If you’re not familiar with the term, page segmentation involves splitting a web page into separate pieces […]

Report: Sitemaps Decrease Crawler Response Time

An SEOMoz post has documented statistics on how submitting a Sitemap to Google and Yahoo impacts the rate of crawling that web site. The results show that submitting a Sitemap to Google and Yahoo decreased the time it took Google and Yahoo to crawl the page. It is important to note that the test was […]

Live Search Testing New Crawler; MSNBot/2.0b

The Live Search Blog announced they are letting a new robot loose. The new search engine crawler is named msnbot/2.0b and will be added to the army of current MSN spiders, currently named msnbot/1.1. The new spider is currently being tested but will ultimately replace the old spider. The new spider will respect the current […]

SEO

Yahoo November Search Update

Yahoo has informed us that they will be rolling out a search update over the weekend. Yahoo said, that you can expect “some changes to our crawling, indexing and ranking algorithms over the next few days and expect the update will be completed soon.” As always, expect the ranking to change, hopefully not too much […]

SEO

How To Make Flash Content Search Friendly

Life is full of tough choices: Mac or PC? Mastercard or Visa? Bud Lite or Miller Lite? And choosing between two good options is never easy, as each has its own merits. Similarly, marketers have been faced with the same kind of tough choice when choosing their website design strategy:  Enhance the website’s appearance with […]

Yahoo Issues Search “Weather Report”

Yahoo has announced another search index update this morning: “We’ll be rolling out some changes to our crawling, indexing and ranking algorithms over the next few days and expect the update will be completed soon. As you know, throughout this process you may see some ranking changes and page shuffling in the index.” Yahoo seems […]

SEO

Google Webmaster Tools Now Provide Source Data For Broken Links

Ever since Google Webmaster Tools started reporting on broken links to a site, webmasters have been asking for the sources of those links. Today, Google has delivered. From Webmaster Tools you can now see the page that each broken link is coming from. This information should be of great help for webmasters in ensuring the […]

How To Walk A Mile In A Search Engine’s Shoes

Small business owners are often curious, and sometimes desperate, to understand why their web sites are doing well — or doing poorly — when it comes to search engine visibility. Online forums and message boards are filled with questions like “Why is my competitor outranking me?”, “Why doesn’t my new product page bring me any […]

Google Says, Don’t Rewrite Dynamic URLs To Static URLs

Almost two years ago, Google updated their guidelines to say that they can now properly handle crawling dynamic URLs. But last night, Google posted a blog entry now telling webmasters not to use rewrite rules for dynamic URLs, at least in most cases. Google say, in bold, “avoid reformatting a dynamic URL to make it […]

SEO

Key Takeaways From SMX Advanced For In-House SEMs

Perhaps for the first time ever I’m guilty of being slack. Either that or I’ve been seriously buried (a condition I’m sure most in-house SEMs are quite familiar with). Although there was some great live blogging of SMX Advanced, nobody to my knowledge summarized key takeaways for in-house search marketers. So here are what I […]

Live Search Webmaster Center Gains Crawl Error And Backlinks Reports

Today, the Microsoft Live Search Webmaster Center is launching its first major update since its initial debut last fall. The Live Search Webmaster Center is Microsoft’s hub for their communication with site owners, and the tools portion of that center now has new crawl issues reports, backlinks data, and download functionality. The Webmaster Center was […]

No, The Toolbar Does Not Lead Google To Index My Content

Matt Cutts of Google has posted another one of his debunking theories post, this time on the Google Toolbar aiding Google in indexing content. Matt said, “if Ken Simpson is implying that the Google Toolbar led to these urls being crawled, then he’s mistaken.” Ken Simpson was paraphrased in an InformationWeek article saying, the “Google […]

The Long Road To The Debate Over “White Hat Cloaking”

I really hate arguments over cloaking. Like really hate them. But Rand Fishkin recently did a chart outlining the degrees of "safeness" to cloaking that in turn riled up Google’s Matt Cutts in comments there. No doubt others at search engines also dislike the idea that any cloaking is "white hat." So I wanted to […]

SEO

Yahoo Search June 2008 Index Update

Yahoo has warned us of an up and coming Yahoo Search index update. Priyank Garg & Sharad Verma of Yahoo Search said Yahoo will “be rolling out some changes to our crawling, indexing and ranking algorithms over the next few days, but expect the update will be completed soon.” The last Yahoo update was just […]

For Google, Add .0 To .EXE As File Extensions To Avoid

SEOmoz discovered that ending your URLs with a .0 will prevent your pages from being found in the Google index. That means we need to add the .0 to the list of file extensions you should avoid using. Stephen Spencer’s News.com article said, “According to Matt, the file extension in your URL won’t affect your […]

SEO

Everything You Wanted To Know About Blocking Search Engines

Last week, the three major search engines came together to say how they agree — and disagree — over the Robots Exclusion Protocol. It’s such an important standard, one every webmaster should understand. To help, Vanessa Fox has compiled an extensive and outstanding overview of it at Jane & Robot in her Managing Robot’s Access […]

SEO

Yahoo!, Google, Microsoft Clarify Robots.txt Support

Today, Google, Yahoo!, and Microsoft have come together to post details of how each of them support robots.txt and the robots meta tag. While their posts use terms like “collaboration” and “working together,” they haven’t joined together to implement a new standard (as they did with sitemaps.org). Rather, they are simply making a joint stand […]

10 Fundamental Tips To Improve Your SEO

In my colleague Matt Lester’s recent Search Engine Land column, he discussed ten tips for a more effective paid search campaign. For this article, I’ll follow up Matt’s advice with ten tips to help you develop a more effective search engine optimization (SEO) campaign. But before we dive into the tips, let’s briefly look at […]

Live Search News: MSNBot, Products Launches And Deep Links Appear

Microsoft’s MSNBot is doing more crawling these days and is alerting webmasters accordingly to revisit robots.txt files. More detail can be found on the Live Search Webmaster blog. In other news, Live Search blogs officially talked about its version of sitelinks that first appeared back in September (and see here).

SEO

New Yahoo Search Weather Report

Yahoo Search says it’s rolling out a new search update. Yahoo’s post says the company will be “rolling out some changes to our crawling, indexing and ranking algorithms. While we expect the update will be completed soon, as you know, throughout this process you may see some ranking changes and page shuffling in the index.” […]

SEO

Yahoo Updates Crawler: Introduces Yahoo Slurp 3.0

Yahoo has announced the launch of their new crawler, Yahoo! Slurp 3.0. 3.0 is slowly being pushed out over the course of the “next several weeks,” including “some infrastructure updates,” Yahoo said. I suspect the recent changes we reported with the Yahoo Search Tax Day update is related to the “infrastructure updates.” The new spider […]

Google Now Fills Out Forms & Crawls Results

One of the biggest search challenges has long been that the major search engines like Google cannot crawl material that can only be retrieved through the use of forms. Now Google is filling out those form to obtain the information previously hidden, the company has announced.

SEO

How Flash Can Be Search Engine Friendly

For a long time there has been a tug of war between developers and designers who love to use Flash and SEOs who know how bad Flash can be from a search engine crawler perspective. The designer wants the beauty of an all Flash site. The SEO wants to bring the site traffic. It’s a […]

Google Offers Robots.txt Generator

Google’s rolled out a new tool at Google Webmaster Central, a robots.txt generator. It’s designed to allow site owners to easily create a robots.txt file, one of the two main ways (along with the meta robots tag) to prevent search engines from indexing content. Robots.txt generators aren’t new. You can find many of them out […]

You’d Be Wise To “NoFollow” This Dubious SEO Advice

For the past few months, I have been listening to some of my colleagues talk about the nofollow attribute and how to use it to sculpt a page’s PageRank. I heard this SEO advice first at SMX in Stockholm and most recently at SMX in Santa Clara. Stephan Spencer wrote about it in a recent […]

SEO

Yahoo Search March ’08 Weather Report

Yahoo Search sent me a note that tomorrow they will be rolling out a new update. Yahoo told me that they made “some changes to their crawling, indexing and ranking algorithms.” The update is expected to be a quick one, but you may notice “ranking changes and page shuffling in the index,” according to Yahoo. […]

SEO

Sitemaps.org Update: You Can Now Store Your XML Sitemap Files Anywhere!

The major search engines have announced an update to the sitemaps.org protocol which enables site owners to store their XML Sitemap files in any location — even on a different domain than the one referenced in the Sitemap. This will be a welcome change for those who manage multiple domains and would like to keep […]