Many SEOs and SEMs have been noticing updates at Google, Yahoo and Ask.com this past week. I thought I catch you up on what has been noticed at each engine.

On the Google front, as reported at the Search Engine Roundtable, many webmasters noticed their PageRank scores drop. After a lot of discussion, concern and debate, Matt Cutts of Google commented explaining that the PageRank scores are fluctuating due to the canonicalization updates Google is pushing through. He said, as long as you are not selling links, “Don’t Panic.” These things will work themselves out and your rankings should not be impacted.

Google is not the only search engine people are taking notice of recently. I reported that webmasters are also noticing an update at Yahoo and even at Ask.com. Yahoo has not announced a “weather report” at the main Yahoo Search Blog, they have however announced it at the Japanese blog. Many are of the belief that Yahoo has done some sort of update and it is impacting search referrals for many. Many have also noticed that Ask.com did some sort of test update. This Ask update does not seem to be related to the super vertical push they are making, but rather an influx of Google ads into the listings. It is hard to say for sure, because only some are part of this test.

In addition, Google Blog Search continues to adjust their algorithms and indexing techniques. Some are reporting that Google Blog Search is failing to index their new blog posts, while others continue to complain about how they adapted to index the full text of the blog post.

That is a quick snapshot of what took place on the SEO front this week.

The paid links debate is back, this time about whether Google wants all links in a paid post to have a nofollow attribute. Below, a look at the latest round, plus a recap of this year’s "War On Paid Links" by Google and where the other search engines stand on the subject.

The current round was sparked by an IZEA (previously Pay Per Post) post inviting the major search engines to clarify their stances on links in paid posts. Ted Murphy blogged that he talked to Matt Cutts at Pubcon, who told him that all links in a paid post should have the nofollow attribute, not just links to the site that paid for the review:

I explained to Matt that in SocialSpark all links required by an advertiser would carry the no-follow tag. I thought this would be a great thing. Matt commended the decision, but then added ALL links inside of any sponsored post should carry the no-follow tag period, regardless of whether they are required, not required or even link to the advertiser paying for the post. That means if Nikon pays me to review a camera and I link off to a site about photography that link needs to be no-follow, along with the link to the blog of my buddy the photographer. His reasoning was that the sponsored post wouldn’t exist without the sponsor paying for it, therefore all the content is commercial and should be no-follow.

The ramifications of that statement and policy didn’t hit me until I was on a jet back to Orlando. Is Google really saying that all content that is commercially driven by a sponsor should carry no-follow tags?

That type of policy seemed a bit much to some people, such as Andy Beard, who blogged a variety of examples of how this type of policy might be interpreted, concluding:

There is absolutely no way I can comply with these current new demands, I would have to stick nofollow on every link within some of my most popular and highly rated content.

You’ll also find discussion of Andy’s post here at Sphinn.

Matt Cutts then responded, explaining that putting nofollow on all links in a paid post would be safe but suggesting he was not saying it was required:

I think quoting me as saying "ALL links inside of any sponsored post should carry the no-follow tag period, regardless of whether they are required, not required or even link to the advertiser paying for the post" is different than our conversation. I believe that I said that adding nofollow to all links in paid posts would certainly be safe. Then I asked if you were going to require nofollow on required links, why not put them on all links in paid posts? I think you replied that your business model didn’t support that, but I may be misremembering.

Ted then did a fresh post, accusing Google of having a double-standard over paid links, since TechCrunch — which had been previously held up as an example by IZEA of not having sponsored links without the nofollow attribute – was not ever apparently penalized. TechCrunch got fresh attention since it recently just added nofollow (see posts on this from Andy Beard, Michael Gray, and discussion at Sphinn). Wrote Ted:

Now, almost a month later TC decides to add a no-follow to their most recent thank our sponsors post and you commend them in your comment. You were clearly aware of the situation. You said it was a violation. Why didn’t TC suffer the same punishment as the smaller bloggers that were hit with a PR0? Why is there a double standard? What about the previous thank our sponsors posts that still don’t have no-follow?

It is this double standard that makes it very difficult for us to enforce policies on linking. Competing businesses are not held to the same standard.

The debate is just the latest in a long line of disagreements over how search engines treat paid links. A quick search of Matt Cutts’s blog shows that he’s been talking about it a lot, and for a long time.

Notably, Matt was talking about the issue back in 2005, and in January 2006, said he thought the topic was pretty "picked over". But in April 2007, Matt blogged about how to report paid links, which Danny felt — in his Search Engine Land: Time For Google To Give Up The Fight Against Paid Links? post — kicked off Google’s second "war on paid links," with the first having been over the SearchKing case.

If it was a second war, things continued along. Google launched a paid links reporting form within Google’s Webmaster Tools in June. At SES San Jose, Matt participated in the "Are Paid Links Evil" session, where Michael Gray proclaimed that "Google is not the government." For more, see:

• Matt Cutts’s comments and slides from SES
• Michael Gray’s slides from SES
• SEOmoz: The Paid Links Debate Rages On
• Search Engine Roundtable: Are Paid Links Evil?

During the fall, some sites saw their Google Toolbar PageRank drop. First sites that were believed to be selling links saw a PageRank reduction, kicking off a huge debate across various search blogs. Soon after, sites that had a substantial number of links from link-selling sites saw drops. Even though those links weren’t paid, the many link-selling sites either had less PageRank flowing from them or lost their ability to pass PageRank altogether, which created a ripple PageRank reduction effect, which Matt and I discussed in this WebProNews video.

Finally, Google formally added a warning against link selling to its online help files in late November, and then in early December the Official Google Webmaster Central blog featured a comprehensive post on Google’s stance on the subject.

With Google’s stance on buying and selling links pretty clear now, how about the other major search engines? According to Google, they’re in agreement. On the Google Webmaster Central blog, Matt posted:

Q: Is this a Google-only issue?

A: No. All the major search engines have opposed buying and selling links that affect search engines. For the Forbes article, Google Purges The Payola, Andy Greenberg asked other search engines about their policies, and the results were unanimous. From the story:

Search engines hate this kind of paid-for popularity. Google’s Webmaster guidelines ban buying links just to pump search rankings. Other search engines including Ask, MSN, and Yahoo!, which mimic Google’s link-based search rankings, also discourage buying and selling links.

Search Engine Land wanted to hear from the other search engines first hand, however. Immediately after that blog post, Barry Schwartz asked them all for an official stance on these three questions:

1. What is your policy on buying paid links for ranking purposes. If you buy, what might you do?  
2. What is your policy on selling paid links for ranking purposes. If you sell, what might happen to you? 
3. And if you sell paid links, do you recommend using nofollow or routing through robots.txt?

To date, Microsoft and Yahoo have failed to respond despite follow-up requests. (Updated January 9: Microsoft has responded. See below.)

Ask.com said:

Trading links is a common practice on the Internet. Our primary focus is to distinguish high quality links from low quality ones regardless of whether they are paid or organic. We are not interested in penalizing sites that buy or sell links as long as the links are relevant and useful for searchers. And, we believe the ExpertRank algorithm is optimized to help identify quality links from those that would not contribute to the end-user experience.

While Google says you shouldn’t buy or sell links unless you use some type of link credit blocking mechanism like nofollow, Ask doesn’t care.

While we still await a direct response from Microsoft, We have received an official response from Microsoft, and there are other comments out there saying they don’t like paid links, though requirements for blocking these with nofollow are not mandated in those:

Earlier this month, Nathan Buggia of Microsoft’s Live Search Webmaster Center talked about paid links in a SEOmoz WhiteBoard Friday video and later commented in print:

Whether it is pay-per-post, paid links, or product placement in movies, if you’re not telling your audience you’re mixing in a dash of paid content into their organic soup, I think that it is eventually going to back fire.

Live Search’s Eytan Seidman recently said:

The reality is that most paid links are a.) obviously not objective and b.) very often irrelevant. If you are asking about those then the answer is absolutely there is a risk. We will not tolerate bogus links that add little value to the user experience and are effectively trying to game the system.

Ramez Naan of Microsoft’s Live Search has responded to our query with the following:

We think of links as a signal to ranking in as much as they reflect actual value to an end user. A link that is white text on white is obviously not valuable to the user, and if we detect such techniques we may disregard the link and may penalize the page it’s on. Paid links are a gray area. Are they of value to the end user? Sometimes they are. Often they’re less valuable and less relevant than the organic links on a page. We reserve the right to treat them that way.

It’s important for webmasters to keep in mind that search algorithms are constantly evolving. Given that, you should think more about the principles behind our ranking choices than the specific implementations we may have today. The core principle is: reward content and links that are valuable to the user. If you violate this principle, it may work out for you in the short term, but as algorithms get smarter it will work less and less, and may even backfire.

A second principle is: manipulating our algorithms in ways that does not add value to the end user is bad. If we detect such manipulation, we may disregard it, and may even penalize you. And again, our techniques for detecting manipulation improve every day.

As for Yahoo, it came out in support of nofollow when originally introduced, although the post specifically talked about comment spam, not paid links.

We’ll try again to get answers from the others (as of January 9th, only Yahoo! hasn’t replied) on our three questions. This being New Year’s Eve, we don’t expect they’ll come today, but hopefully they will in the near future.

If there’s anything I particularly hate when it comes to SEO, it’s the meta keywords tag. I so wish it had never been invented. It’s practically useless, yet people still obsess over it. In this article, I’ll explain more about why you shouldn’t worry about it except perhaps for misspellings, as well as which search engines support it.

The meta keywords tag is one of several of meta tags that you can insert into your web pages to provide search engines with information about your pages that isn’t visible on the page itself. For example, my Meta Robots Tag 101: Blocking Spiders, Cached Pages & More article covers how you can use a different meta tag — the meta robots tag — to block pages from being indexed. Users don’t see this information (unless they look at your source code), but search engines do.

Meta Tags & Your Header

Meta tags go within the header area of your web pages. A typical head might look like this:

<head> <title>Welcome To Shoe Central!</title> <meta name=”description” content=”All the best prices on shoes!” /> <meta name=”robots” content=”noodp” /> <meta name=”keywords” content=”shoe, shoes, shoee, shos, footwear” /> </head>

The header is the section that begins <head> and ends </head>. Between those elements, in our example, you have these tags:

• Title: The text here becomes the title that is shown in search engine listings, in most cases.
• Description: The text here is text that search engines sometimes use as a description for your web page when listing it (a meta tag lesson for another time).
• Robots: This particular tag is configured to ensure that the page isn’t described using the a description that the Open Directory might have for it (Meta Robots Tag 101 explains this more).
• Keywords: This tag is the topic of this article, so read on!

History Of Meta Keywords

I’ve long written about search engines and meta tags, but I have never been able to pin down exactly who created the meta keywords tag. There’s a December 1995 internet draft memo that’s the earliest and most authoritative mention of the tag I know of. It says:

<META HTTP-EQUIV= “Keywords” CONTENT= “Italy Product, Italy Tourism”>

The spaces between a comma and a word or vice versa are ignored….

These ‘keywords’ were specifically conceived for exhaustively and completely catalogue the HTML document. This allows the software agents to index at best your own document. To do a preliminary indexing, it’s important to use at least the http-equiv meta-tag “keywords”.

Sounds good, right? Like this is designed for the search engines to use? The issue is that HTML specs like these (especially drafts) are not necessarily used by the search engines. They can use them, ignore them or build upon them as they see fit.

As it turns out, several of the major search engines got together in May 1996 to talk about meta data. That meeting gave birth to a common standard for the meta robots and the meta description tags. As for the meta keywords tag, it was discussed, but no specification emerged.

Despite no specification, both Infoseek (later Go.com, these days no longer crawling the web) and AltaVista (now owned and powered by Yahoo) offered support for the meta keywords tag in 1996. If you looked at their help files at the time, they encouraged site owners to use the tag. Inktomi (now owned by Yahoo) also provided support when it began operations later in 1996, and Lycos (no longer crawling the web) added support in 1997.

That year — 1997 — was the last year that the meta keywords tag enjoyed support among the majority of major crawlers out there (4 out of 7 – Excite, WebCrawler and Northern Light, also crawling the web that year, did not support it).

Support Dies Off

When new search engines emerged in 1998, such as Google and FAST, they didn’t support the tag. The reason was simple. By that time, search engines had learned that some webmasters would “stuff” the same word over and over into the meta keywords tag, as a way of trying to rank better. At the time, search engines didn’t rely so heavily on link analysis, so page stuffing like this was more effective. Alternatively, some site owners would insert words that they weren’t relevant for.

In July 2002, AltaVista dropped its support of the tag. That left Inktomi as the only major crawler still supporting it, causing me to somewhat famously in the SEO world to declare the tag dead, since it was no longer a major ranking factor for even Inktomi:

Traffick.com’s Andrew Goodman wrote recently in an essay about meta tags, “If somebody would just declare the end of the metatag era, full stop, it would make it easier on everyone.”

I’m happy to oblige, at least in the case of the meta keywords tag. Now supported by only one major crawler-based search engine — Inktomi — the value of adding meta keywords tags to pages seems little worth the time. In my opinion, the meta keywords tag is dead, dead, dead. And like Andrew, good riddance, I say!

Since that time, Inktomi was rolled up into Yahoo, which continues to support the meta keywords tag as part of its Yahoo search engine. Or does it?

Search Engine Rep Confusion

Last month, I moderated my last “Meet The Crawlers” panel for the Search Engine Strategies conference series (Goodbye Search Engine Strategies! explains more about my shift from SES to my own SMX: Search Marketing Expo series). That perennial favorite question came up during the session. Who supports the meta keywords tag?

Sigh. But if this question still coming up wasn’t depressing enough, then the search engine reps starting responding with a load of confusion. To paraphrase:

No, we don’t support it. Well, we read it. We read it, but it doesn’t matter. Actually, maybe we don’t read it.

Even Evan Roseman from Google said at one point that Google reads the meta keywords tag, suggesting no doubt to some that Google uses the tag.

To be clear, Google doesn’t. I’ll prove it further below, but it doesn’t, OK?

I gave Evan (hopefully) some good humored hassle afterward for saying this. He’s at least the second Google rep to declare this on panels I’ve moderated in as many years, and the problem is that the engineers (from any of the search engines) often take the question too literally.

Indexing Versus Retrieval Versus Ranking

To understand, let me talk about three different things a search engine does when it crawls and lists your page:

• Indexing: This is where the search engine effectively makes a copy of your page. The search engine is going to read and store the HTML content it finds — all of it. Evan was right when he said that the meta keyword tag is indexed by Google. Google knows that the tag exists and has recorded what’s in it. But that doesn’t mean it does anything else with it.
• Retrieval: This is where the search engine finds all the matching documents relevant for what you searched for. Most of those documents will actually have the words you searched for on them, in the sections that the search engine searches against (there are some exceptions, such as when anchor text is used to find pages. Google Now Reporting Anchor Text Phrases, Google Kills Bush’s Miserable Failure Search & Other Google Bombs and Google Declares Stephen Colbert As Greatest Living American explain more about this). While the search engine has recorded the entire page, it won’t search against everything indexed for retrieval. In other words, Google will look to see if words you searched for appear in the body area of a document, but it will NOT look in the meta keywords tag for matching words. The keywords tag, while indexed, is not used for retrieval at Google. At Yahoo, it is.
• Ranking: This is where the search engine looks at all those documents retrieved for a search and puts them in order of most importance, according to its algorithm. Retrieval (or what information research professionals call “recall”) is about finding everything). Ranking (or what the IR folks call “precision” — see Tim Bray’s excellent On Search: Precision and Recall document) is about getting the best stuff up to the top. Yahoo, while using the tag for retrieval, really doesn’t assign much weight to it for ranking.

Testing For Retrieval

Back to my panel experience. Since the reps were unclear, I declared to the audience that I’d just go out and test it again myself. It’s literally been about five years since I’ve last tested the tag, because I (and many others) feel it is so useless. There are better things to do with our time. But since that question needs a big old stake to the heart, I rolled up my sleeves and got cracking.

On the Search Engine Land home page, I inserted this meta keywords tag:

<meta name=”keywords” content=”qiskodslajdmnkd, ddakaieciuaj jkdalladpaoaw, wdaopeqndlkakljad” />

I had searched for all of these words on the four major search engines of Google, Yahoo, Microsoft and Ask and found no pages that matched. If these search engines made use of the meta keywords tag, I’d know in short order, if my page started coming up.

The tag went up on August 28. I then needed to wait until I could see each search engine had the most current version of my page (Squeezing The Search Loaf: Finding Search Engine Freshness & Crawl Dates explains more on how to do this).

Google: No

It took two days, until August 30, for Google to show the latest version of my page in its index. I searched for each of the words, and my home page didn’t come up. The meta keyword tag was not used for retrieval and thus not supported.

Microsoft Live: No

It took five days, until September 2, for Microsoft to show a version of my page with the meta keywords tag on it. As an aside, Microsoft is kind of annoying. It will say something like this in the cached copy of the page:

This is a version of http://searchengineland.com/ as it looked when our crawler examined the site on 9/2/2007. The page you see below is the version in our index that was used to rank this page in the results to your recent query. This is not necessarily the most recent version of the page – to see the most recent version of this page, visit the page on the web.

If you glance quickly at the date, you might think the page has been revisited fairly recently. But as the text explains, it might be older. Indeed, when I looked on September 2 (as is the case today), the copy of the page in the index was as of August 30, as I could tell from the stories shown.

As with Google, I searched for each of the words, and my page didn’t come up. The meta keyword tag was NOT used for retrieval and thus not supported.

Yahoo: Yes

It took two days, until August 30, for Yahoo to have my latest page. Searches there did bring up the home page for all words. So the meta keywords tag IS used for retrieval.

Ask: Yes

Ask took the longest to show the most current version of my page, not reflecting the changes until today. Actually, when I look at the cached copy even now, it says that the page is from August 13 and uses a redirection URL rather than my http://searchengineland.com address.

Still, I can tell Ask has a version with the meta keywords tag on it since I’m getting back my home page when searching for words in that tag. As with Yahoo, the meta keywords tag IS used for retrieval.

Should You Use It? Sure, For Misspellings

So there you have it — half of the major crawlers (Yahoo & Ask.com) DO support the tag. Should you begin using it? My advice would be only for misspellings and really unusual words.

As explained, the tag can help with retrieval. A word in the tag is treated as if it were a word visible on the page itself. Now that’s handy for misspellings. For example, say you’re writing about Basset hounds. You suspect some people might misspell the name as Bassett hounds, adding an extra T. You could misspell the word yourself on the visible page, but that makes you look bad. You could insert the word and then try to hide it using CSS styles or putting it in the same color as the page background. But this type of “hidden” text is generally against search engine guidelines.

Enter the meta keywords tag. Just do this:

<meta name=”keywords” content=”bassett” />

Now you’ve got the misspelling on your page in a “legal” means that will be read by Yahoo and Ask. You’re still out of luck for Google and Live.com, but two out four ain’t bad.

But I Want To Rank!

What about ranking better with the tag. I mentioned already that many experienced SEOs don’t find it useful. Believe me, if just putting a single word into that tag was going to rank your page better, everyone would be doing it. Instead, search for anything on Yahoo or Ask. You’ll see plenty of pages ranking well for words without those words appearing in the meta keywords tag. And if you do see the words in the tag, it’s more due to coincidence — the words also appear in the body copy, in the title tag and often in links pointing at the page. The words in the meta keywords tag aren’t the primary reason the page is ranking well. Promise.

Back to our Basset Hound example. Sure, you can add the correct spelling to your meta keywords tag. Go ahead, if you want. Just understand that it is not likely to make you rank any better than if you didn’t include it at all. Moreover, beginners are especially likely to spend far too long worrying about getting the “right” words in the meta keywords tag rather than just writing good body copy.

Comma Conundrum

One of the most common questions I used to get way back in the old days was over using commas in the meta keywords tag. Consider these options:

1. <meta name=”keywords” content=”bassett, hound, hounds, basset” />
2. <meta name=”keywords” content=”bassett,hound,hounds,basset” />
3. <meta name=”keywords” content=”bassett hound, bassett hounds, basset hound, basset hound” />
4. <meta name=”keywords” content=”bassett hound,bassett hounds,basset hound,basset hound” />
5. <meta name=”keywords” content=”bassett hound bassett hounds basset hound basset hound” />
6. <meta name=”keywords” content=”bassett hound basset hounds” />

Sigh. See why I hate this tag so much, when I’ve had to deal with people wondering about commas and spaces and variations like this. Let’s take it from the top, as to the motivations behind these versions:

1. This is someone who thinks that each word should be on its own, separated by a comma and with a space in front of the next word.
2. This is someone who thinks that getting rid of the spaces means they can squeeze in more words.
3. This is someone who thinks that if there are particular phrases they want to be found for, those phrases should be together and set off by commas.
4. As with three, but losing the spaces to squeeze in more words.
5. Similar to three but thinking you don’t need commas at all.
6. This is Mr. or Ms. Paranoid. They’re concerned about saying any word too often. So they lose the commas, restrict repetition and hope that proximity will help (IE, put “basset” behind “hound” rather than in front and maybe you’ll still show up for “basset hound.”

Which way should you go? I’d suggest number three, for these reasons:

• Yahoo has long recommended using commas and in particular supported them as a way to separate out distinct terms for those in their paid inclusion programs. I’ll update this page with the latest advice, but commas still seem to make sense.
• Spaces just make things look nicer, and you shouldn’t be shoving a ton of terms in the tag anyway. How long is too long? No idea! In the past, the search engines just wouldn’t index content beyond around 250 to 1,000 characters. Maybe I’ll test this in the future.
• You do want phrases kept together. “bassett, hound” is probably going to be seen as “bassett hound” anyway, but why risk it?

Other Uses

I mentioned that misspellings were a key use for the tag. You could also use it for synonyms. For example, if you have a page all about shoes and you never say “footwear,” you could put that word in your tag. However, it’s far better if you just find a way to make use of the word in the body copy itself. That text is retrieved by all the major search engines, not just some.

Aside from synonyms, perhaps you have a page that’s all Flash or all images. Use the meta keywords tag to describe the page. Just remember that you’re still not likely to rank better than other pages that have textual information. Search engines are textual creatures. Give them what they want.

Some Official Guidelines

The W3C has guidelines (and here) in HTML 4.0 about meta data and search engines, while the XHTML specs don’t get into it at all. Ignore the specs. YES, IGNORE THE SPECS. Some of them are wrong; some are outdated. The only thing I can see that they explain is the difference between these:

• <meta name=”keywords” content=”bassett”>
• <meta name=”keywords” content=”bassett” />

See how the second tag ends /> rather than > in the first? As best I can tell, this is because a meta tag is an “empty element” in XHTML, where there’s not a “start” and a “finish” (as with a paragraph element: <p> is the beginning, with </p> the end). Empty elements in XHTML need that /> format.

I haven’t tested things without the />, but there are so many (so very, very many) pages out there not following that syntax that it is virtually certain Yahoo and Ask will read the tag either way. Doing it fresh? Do it /> style. But don’t go back and start changing things.

Aside from that, if you want to know how a search engine deals with meta data officially, you go to the search engine itself. Ask’s webmaster guidelines don’t mention the meta keywords tag, so that leaves Yahoo:

• Yahoo Quality Guidelines: “Metadata (including title and description) that accurately describes the contents of a web page.” This is telling you don’t lie with your keywords. Don’t insert words that aren’t somehow related to the topic of your page.
• How do I improve the ranking of my web site in the search results?: “Use a ‘keyword’ meta-tag to list key words for the document. Use a distinct list of keywords that relate to the specific page on your site instead of using one broad set of keywords for every page.” Note that it doesn’t say you’ll automatically rank better by doing this. Also, unique words for each page would be my advice, as well — but do NOT worry if you decide to use the same set of key terms on each of your pages. It isn’t that big of a deal.

Looking for the exact format that you should use for the meta keywords tag from Yahoo? You know, commas, spaces and all that. Sorry — they don’t provide it, which is another sign you’re probably worrying too much about it.

Freaked? Skip It

Overall, here’s the best advice I can offer anyone dealing with this tag. If you begin to feel confused, concern, tired or uncertain when pondering it, SKIP THE TAG ENTIRELY. It’s not going to hurt you to not have it, and it’s not worth the time fretting about it.

Jim Lanzone, CEO of Ask.com, has confirmed with me that Ask.com is working on Edison. Edison is the code name behind merging of Ask.com’s two different search technologies they own, Teoma and Direct Hit. The name comes from inventor Thomas Edison, who worked out of New Jersey where’s Ask’s Teoma technology came from and where it still maintains a research base.

Jim would not give me more information beyond that. I did speak with Apostolos Gerasoulis who leaked this information this morning, where he explained that they will be using Direct Hit’s click popularity technology with Teoma’s “subject-specific” link analysis to provide a more social search service.

Apostolos Gerasoulis is very excited about this new algorithm and he told me he feels this will be the most powerful algorithm in search.

Update: I have received a statement from Rahul Lahiri, Vice President of Product Management and Search Technology at Ask.com:

Edison is still in development, so we can’t say too much at this juncture. I can tell you that it’s a next generation algorithm that, among many other things, synthesizes modernized versions of Teoma and DirectHit technologies, as AG said this morning. It’s much more complicated than saying we’re just counting clicks, in the case of DirectHit. The technologies we have, and the patents we hold, go way beyond that. We’re also taking a deeper look at communities and calculating the authorities in those communities. We were really inspired by looking into the universe of user behavior, and what that could tell us, and the social fabric of the Web itself, and what that tells us. We’re also rolling out an upgraded search infrastructure over the course of 2007 and building a new datacenter along the Columbia River in eastern Washington, which will help our speed, freshness and data quality. It’s safe to say that Edison itself will roll out over the course of the year, as we improve it and tweak parameters.

Last November, Google, Microsoft and Yahoo united to support sitemaps, a standardized method of submitting web pages through feeds to the search engines. Today, the three are now joined by Ask.com in supporting the system and an extension of it called autodiscovery. This is where the major search engines will automatically locate your sitemaps file if the location is listed in a robots.txt file. Announcements are up from Google and Ask now Yahoo and Microsoft.

Information on how to create sitemaps files can be found at the Sitemaps.org site. Aside from the sitemaps XML formal, you can also provide RSS 2.0 or Atom 0.3 or 1.0 feeds. That’s handy for those with blogs that already generate these feeds.

Sitemaps XML files too complicated? Don’t run a blog? Note that the site has newly expanded information on how you can submit a simple list of URLs in a text file.

In the past, if you created a sitemaps file, you then had to manually tell the search engines where to find it. With today’s announcement, search engines will check your robots.txt file for a link to a sitemaps file, then get the file from that location. This is a big plus because all the major search engines regularly check robots.txt files as part of their ordinary crawling.

To add the location, just put a line like this anywhere in your robots.txt file:

Sitemap: LOCATION-OF-SITEMAPS-FILE

Replace the LOCATION-OF-SITEMAPS-FILE with the actual location. For example, if you ran a site at mywonderfulsite.com and had a sitemaps file called allmypages.xml in your top level, the reference would be like this:

Sitemap: http://mywonderfulsite.com/allmypages.xml

Have more than one sitemaps file? Ideally, you’d create a special "sitemaps index" file that links to all of them, then put a link to the sitemaps index file in your robots.txt file. If that sounds like too much work, you can have more than one sitemaps URL listed in the robots.txt file.

Aside from autodiscovery, you can also ping Google and Yahoo with the location of your file. The Sitemaps.org site has more instructions on this in general. For specifics:

• Google: See here. Note that this pinging is different than the pinging Google also supports for blog search.
   
• Yahoo: See here and here. Unlike Google, the same pinging system is used for both web and blog search, to my understanding.

Both Google and Yahoo also allow you to manually submit sitemaps files. In both cases, doing this via their Google Webmaster Central or Yahoo Site Explorer systems gives you access to specialized monitoring and reporting tools or information on how they crawl you.

For more about these tools, or how each individual search engine handles sitemaps files, please see the links below:

• Google: Google Webmaster Central or sitemaps info.
• Yahoo: Yahoo Site Explorer or sitemaps info.
• Microsoft Live: submit page or submit help info
• Ask: webmaster help info

Keep in mind that Microsoft and Ask are still lacking references to sitemaps information, but I expect this will change over time.

For related coverage, see here and here on Techmeme.

In a comment left at the Stepforth blog, Jim Lanzone, Ask.com’s CEO, has said that he considers paid inclusion a “dis-service”. Jim explains that he finds it “hypocritical to charge for something we need to do anyway,” i.e. crawl the web to find quality pages to present to the searcher. In addition, Mr. Lanzone explains that it “blurs the line” between “paid content and editorial content” making it hard for the searcher to possibly trust the search results.

Lanzone does not go as far to accuse Yahoo of skewing the results in favor of paid inclusion results. Lanzone does say that “Paid Inclusion is just not on the table for Ask.com,” which is good to know.

I have confirmed from Ask.com that these comments are from Jim Lanzone, the CEO of Ask.com. Ask.com discontinued its paid inclusion service on August 31, 2004.

Last week, I covered a new command for the meta robots tag — one to prevent search engines from using Yahoo titles and descriptions. In doing that, a number of questions came up about the meta robots tag syntax itself. Google Webmaster Central has now posted “Using the robots meta tag,” providing some clarity from Google. In addition, both Yahoo and Microsoft have also sent me information on using the tag. I’ll run through what everyone says below, complete with charts for easy at-a-glance comparisons.

The meta robots tag was an open standard created over a decade ago and designed initially to allow page authors to prevent page indexing. Over the years, various search engines have added additional support to the tag.

Let me start off by saying that if you DO want your pages in search engines, then DO NOT use the tag. By default, the major search engines will index any page they find. Yes, there is a form of the meta robots tag you can use to explicitly tell search engines to index your pages. It looks like this:

<meta name=”robots” content=”index”>

There’s also a form you can use that adds the command “follow,” which tells the search engines to index your page and also follow any links they find on that page to other pages, which they can then index. It looks like this

<meta name=”robots” content=”index,follow”>

You do NOT need to use either form if you DO want your pages in the search engines. Without either form, they’ll naturally index your pages and follow your links. That’s what they do.

I always joke that putting these forms of the meta robots tag on your web pages is like putting a Post-It note on your chest that says “breathe.” Hey, if you forget to look at that note, you’ll still breathe. That’s what you do, by default. And that’s what the major search engines do. By default, they inhale web pages without you putting up a meta tag telling them to do so.

Now if you DO NOT want your pages in a search engine, then it’s time to perhaps break out the meta robots tag, if for some reason the robots.txt alternative isn’t suitable. Want to keep a particular page out? Then put this on that page:

<meta name=”robots” content=”noindex”>

See the “noindex” value? That tells the search engines that see this page not to include them in their listings. Remember — as I explained before — this will not prevent the pages from being spidered. That’s because search engines have to keep revisiting the page in order to see if the tag is removed. The tag only keeps the page out. Here’s my earlier chart on that topic.

What if you don’t want links followed? Sure, you can do this:

<meta name=”robots” content=”noindex,nofollow”>

That extra command, “nofollow,” tells the search engines not to follow any links on that page. Google recently covered this more as an option. But as Google also explained, links from a page with this tag might still get crawled. That’s because if anyone else links to a particular page WITHOUT a nofollow value, then the search engine will follow that link.

So far, I’ve covered all the commands that were originally created with the tag back in May 1996. Since then, more commands (also called values or attributed) have been added. For example, Google writes today to summarize several options you can use. Quoting Google:

• NOINDEX – prevents the page from being included in the index.
• NOFOLLOW – prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
• NOARCHIVE – prevents a cached copy of this page from being available in the search results.
• NOSNIPPET – prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
• NOODP – blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.

At times, you may want to use more than one of these commands. I’ll get back to that. But first, how about another chart? I’ll cover the major commands you may want to use below:

Several of these are already explained above, in what I quoted from Google. They work the same way for the other major search engines. I’ve also linked to help information from each search engine for more specific advice.

The NOYDIR command is fully explained in my previous Yahoo Provides NOYDIR Opt-Out Of Yahoo Directory Titles & Descriptions post. Only Yahoo supports this, but none of the other major search engines used Yahoo titles and descriptions for listings, so it doesn’t really matter for them.

Now on to the topic of a meta robots tag having multiple values. What if you wanted to keep a page from being cached by all the major search engines and also ensure that neither Open Directory or Yahoo Directory descriptions are used. First, you need the values of the commands to say this. From the table above, they are:

• NOARCHIVE
• NOODP
• NOYDIR

Next, you need to decide what robots to target. We’ll keep it simple for now. To target ALL robots, you use this value:

• ROBOTS

Now to the meta robots format. Without the values, it looks like this:

<meta name=”NAME-OF-ROBOTS-TO-TARGET” content=”COMMANDS”>

We replace that NAME-OF-ROBOTS-TO-TARGET part with the name of the robots we’re, well, targeting. As explained, that’s ROBOTS, in order to target them all. I’ll put it in bold below:

<meta name=”ROBOTS” content=”COMMANDS”>

Now we put in the commands we want to tell the robots, each separated by a command. The order doesn’t matter. Again, I’ll bold the commands:

<meta name=”ROBOTS” content=”NOARCHIVE,NOODP,NOYDIR“>

Voila! Put that tag ANYWHERE inside the header area of a web page like this:

<HEAD> <meta name=”ROBOTS” content=”NOARCHIVE,NOODP,NOYDIR”> </HEAD>

Then you will be telling all major search engines not to cache the page, nor to use Open Directory or Yahoo Directory titles or descriptions for you page listings.

Notice that in the tag above, there are no spaces between the commands. What if I did this?

<meta name=”ROBOTS” content=”NOARCHIVE, NOODP, NOYDIR”>

Google writes today that spaces make no difference. Use them if you want or not, the tag means the same thing. Microsoft tells me the same thing, as does Yahoo.

What if you did this, with no commas:

<meta name=”ROBOTS” content=”NOARCHIVE NOODP NOYDIR”>

Microsoft tells me this is fine. I didn’t ask Yahoo about this, and Google says commas MUST be used. So use commas and don’t be a pain.

Now what if you want to tell search engine different things. Maybe you want Microsoft not to use the ODP descriptions, Google not to cache pages, Yahoo not to follow links on a page and Ask not to index the page at all. Maybe you want to get your head examined for being so strange, too. But aside from your mental health, it is possible to do all this.

You need to have a robots tag for each particular search engine you want to target. See that chart above? At the bottom there’s a “Robot Name” row. That shows you the name of each search engine’s “robot” or “spider” that you’ll issue a command to. With the robot names, we then give each of them their specific commands:

<meta name=”TEOMA” content=”NOINDEX”> <meta name=”GOOGLEBOT” content=”NOARCHIVE”> <meta name=”MSNBOT” content=”NOODP”> <meta name=”SLURP” content=”NOFOLLOW”>

You could also tell all robots to do one thing — say not to follow links — while also issuing a second robots-specific command such as telling only Google not to cache the page:

<meta name=”ROBOTS” content=”NOFOLLOW”> <meta name=”GOOGLEBOT” content=”NOARCHIVE”>

But wouldn’t a search engine only follow the specific tag written for it? In other words, if you target Google with a specific command in the “GOOGLEBOT” tag, then wouldn’t it follow only that tag and ignore the other?

Google, Microsoft and Yahoo say they will honor them both. I don’t know about Yahoo. That’s why you see “???” in that “Does Robot Specific Tag Override All Robots Tag?” section of the chart above. I’ll try to get that answered.

What if you had more than one “all” robots tag like this:

<meta name=”ROBOTS” content=”NOFOLLOW”> <meta name=”ROBOTS” content=”NOODP”>

As explained, you could easily do this instead:

<meta name=”ROBOTS” content=”NOFOLLOW,NOODP”>

But if for some reason you did do it the other way, Microsoft and Yahoo have told me that’s just fine. They honor the information in BOTH of the robots tags. Google’s post today says the same thing.

Finally, the Google post provides reassurance that capitalization doesn’t make a difference. I’ve shown things in various ways above, sometimes the commands in ALL CAPS, sometimes in lowercase. As Google says, case makes no difference. To quote their post:

Googlebot understands any combination of lowercase and uppercase. So each of these meta tags is interpreted in exactly the same way:

<meta name=”ROBOTS” content=”NOODP”> <meta name=”robots” content=”noodp”> <meta name=”Robots” content=”NoOdp”>

Ah, but what about something like this:

<MeTa nAMe=”RoBots” conTEnt=”NooDP”>

Well, Google didn’t go that far. But my experience over the past decade has been that meta tags are not case sensitive at all with the major search engines. So I think you’re safe in whatever case, for all the major search engines.

A reader emailed me today noticing that Google was showing a date next to his
listing, which made me think this was a good time to revisit how, when
and where search engines show crawl dates for pages. These dates are a useful
way for site owners to understand how often they are being revisited or for
anyone to "squeeze the loaf" of a search engine to see how fresh it is. Here’s a
search engine-by-search engine rundown on date display. I’ll also cover how
we’ve sadly lost crawl dates being embedded next to listings, over the years.
But that’s not all! Read now and you’ll even get a free at-a-glance table
explaining how dates are displayed. Read now — web server operators are
standing by!

Google

When you do a search, some pages may show a date below the description of a
listing, as illustrated below:

I thought Google had long done this for certain pages that it revisits on a
super-frequent basis. And when I did a search for
cars today, I saw a date
like this coming up for the cars.com listing as shown above. An hour later, the date was
gone. I then tried that search again using a particular Google data center,
rather than whatever data center was assigned to my browser randomly. Doing the
same
search at that data center gave me dates again.

I’m checking with Google on how long dates have been showing and why they may
come and go as I saw today. I’ll postscript what I’m told at the end of this
story.

The example above shows that only some pages have dates. In contrast, the
Google Cache can
give you dates for nearly any web page.

The Google Cache allows you to view a copy of a page that is stored on
Google’s servers, rather from the website directly. (Don’t like Google caching this
for your site? Learn how to prevent it

here and

here. Don’t see a cached link option? Then the site owner is blocking
caching).

Going back to our search for
cars and the screenshot
above, you’ll see that the disney.go.com listing doesn’t have a date next to it.
To find the date the page was visited, you have to click on the link that says "Cached" under the description of that
listing. That makes the cached page load like

this. At the top of that page, you’ll see this:

This is Google’s

cache of
http://disney.go.com/disneypictures/cars/ as retrieved on
22 Feb
2007 14:34:08 GMT.

See the date and time, which I’ve put in bold? That’s when the page was last visited by Google.

FYI, before
September 2006, that date reflected the last time Google found the page to have
changed, not when it was last visited. In other words, if Google visited the
page in January 2005, then revisited it throughout the year but the page never
changed, the cached date would keep saying January 2005.

Since September 2006,
that’s been different. The date was altered to reflect the last time Google
visited the page — a good change to make. Google explains more about this on
the Google Webmaster Central blog

here, and Google’s Matt Cutts also did a video about it

here.

The options above allow anyone to see the freshness of any pages within
Google, one page at a time (as long as they are cached). What if you want to get industrial strength
and view the freshness of all your own pages at once?
Unfortunately, the Google Webmaster
Central tools don’t let you see the last time all your pages were spidered.
But that’s something they’re considering for the future. The tools will,
however, show you any problems Google had in reaching any of your pages and the
last time a crawl error happened for those pages. Using the "Crawl rate" option
found under the Diagnostics tab, you can also see a general graph of crawling
activity to your site.

There is one other type of date that you might see associated with
listings that has nothing to do when the page was crawled. Look here:

See the "3 visits – Feb 14" part? That’s coming from
Google
Personalized Search and shows that I’ve clicked on that listing 3 times,
with the last visit being on Feb. 14. My
Google Ramps Up
Personalized Search article from earlier this month explains more about how
Google Personalized Search works and can be disabled, if you don’t like it on,
as now happens much more often.

Microsoft Live Search

Microsoft Live Search operates like
Google. Some pages show dates next to them, as I’ve highlighted below:

As with Google, this seems to happen with pages that are being spidered
frequently, but I’ll check on this. Does a page lack a date? Then click on the
"cached page" link. When the cached page loads, you’ll see something like this
at the top of it:

This is a version of

http://www.pixar.com/theater/trailers/cars/index.html as it looked when
our crawler examined the site on 2/16/2007. The page you see below is the
version in our index that was used to rank this page in the results to your
recent query.

The date (which I’ve but in bold above) tells you when the page was last spidered.

Don’t see a cached page
option? The site owner is probably blocking caching. Are you a site owner that wants to
block caching? Visit the
help area at
Live and
search for "cache" to find more info. I’d point you to the right place, but it
remains impossible to link to particular pages in Microsoft’s absurd help
system.

Ask.com

At Ask.com, you can only get dates by looking at the cached pages,
similar to how that works at Google and Microsoft. Click on the "Cached" link that you’ll see
next to the URL of a listing, as highlighted below:

At the top of the page, you’ll see something like this with the date and time
(shown in bold below) that the page was last visited:

Below is a cache or saved snapshot of 

http://www.cars.com/  as we found it on February 19, 2007 1:24:56 AM.

Yahoo

At Yahoo, you can only get dates one way, through using
Yahoo Site Explorer. You’ll
have to create an account for your web site, then authenticate your account,
then you’ll be shown last crawl dates as I’ve highlighted in the first listing
below:

More than any other search engine, Yahoo makes it easy for a site owner to
see the freshness of many pages all at once. However, the huge disadvantage from a
searcher perspective is that you can’t spot check the freshness of any page you
randomly select.

The Date & Freshness Table

I love nothing more than doing tables, so let’s put everything above into a
nice one:

Ideally, I’d like to see that top row — "Dates Next To Listings?" — be
completely "Yes." Some site owners block caching, which makes it hard to measure
freshness. Putting the dates right next to the listings makes it easy for anyone
who cares to see at a glance if a search engine is stale or fresh.

In fact, I have to laugh. I’ve been asking for this for years. On the old
features chart I used to maintain about dates, I

wrote in 2001:

Along with the page description, some search engines show the date when a web
page was created or modified. As noted above, these dates may not always be
reliable. However, they do provide a useful clue as to how fresh or stale a
search engine’s listings are. Thus, search engines that show a date deserve
praise for doing so.

That was from 2001! Nearly six years later, it’s still the case that dates
aren’t being shown. In fact, it’s a reversal. Back in 2001, the major search
engines of AltaVista, HotBot (Inktomi) and Northern Light all showed dates for
all listings right within search results. Fast forward to today, and none of the
major search engines do.

The reason is simple enough. Over time, the search engines either couldn’t
maintain freshness or didn’t want to show they were sometimes stale. So dates
either went away or never got added. C’mon gang — time to bring them back right
into the search results. If they aren’t there by default, make it an option
people can enable.

Verifying Freshness

In the meantime, there’s a favorite tactic for those search watchers who want
to track freshness. Google’s Matt Cutts
once wrote
about this back in 2005, describing exactly a technique I and others have long
used. You simply find a page that you know carries a date that’s constantly
updated. Look at the cached page and see what the time and date says on it.

But Yahoo doesn’t show a date on cached pages! No, it doesn’t, but you’re not
looking for the date that the search engine inserts. You want the date on the
page itself. For example,

here’s the cached page over at Yahoo for CNN:

See the part I highlighted in red, that says:

UPDATED: 3:53 a.m. EST, February 26, 2007

That’s the date that CNN had on its own page when the Yahoo spider last
visited. When I looked, the date and time was 3:10 pm EST on February 27 — so
the page is only 12 hours old. Not bad in this case, but I wouldn’t expect a
major news site to be much out of date.

Return Of The Freshness Guarantee?

Finally, I’ll leave you with this trip down memory lane. Back in June 1999,
AltaVista once offered a freshness guarantee that was quickly broken. As I
wrote at
the time:

"AltaVista search is able to
make its Freshness Guarantee: no search site will have fresher results than
AltaVista."

AltaVista unveiled its first
"Freshness Guarantee" back when it relaunched in June, promising that its
entire index would be refreshed at least once per month. That guarantee was
almost immediately broken, as even AltaVista President Rod Schrock admitted
when we talked recently. "We turned our attention to this new system," Schrock
said.

OK, fair enough — they wanted
to build something even better. But this new guarantee has already been
broken, as described above. If claims like these are going to be made, then
they should actually be met. And not to meet them in the midst of a huge media
blitz is an incredible blunder.

Freshness is one important component to what makes a good search engine. It’s
not the only thing. Having fresh results means nothing if the results aren’t
relevant. And some pages don’t need to be spidered that often. But putting dates
next to listings is an easy form of search "food" labeling that can give
reassurance about a major search engines. Surely it’s time for dates to make a
comeback.