It’s no secret that newspapers have been struggling with the disruptive innovations introduced by the internet, and this has resulted in some level of accusation towards Google and a circling-of-the-wagons mentality by the industry. But, what if they were to go in the opposite direction, with fuller engagement? Here’s one tactic for how to go about it via SEO.

In the last few years, I’ve visited a lot of newspaper websites for various projects. These sites are most frequently the online arms of what were once strictly printed local newspapers. When visiting these sites, I’ve been struck by the technical clunkiness of most—they’re typified by poor usability, layouts still closely influenced by traditional print newspaper layouts, dysfunctional on-site search engines, and content management systems hamstrung with badly-formed page templates.

Naturally, these sites are not optimized for search engines nor to make their content readily findable via search. It’s unsurprising that the sites are search-unfriendly. The newspapers probably feel highly conflicted in regards to search—the nostalgic desire for successes experienced in the past have made them grow unhappy with the internet paradigm, and they’ve worked each other up into a frenzy to hold Google responsible for their troubles. It’s hard to expressly invite a perceived enemy into your house on one hand while issuing invective against him on the other.

(I have also encountered newspaper sites which have optimized by some degree. But, these seem fairly few, and even some of them have only taken faltering steps in that direction. The exceptions are some of the biggest players such as the New York Times, Chicago Tribune, Wall Street Journal, Washington Post and other juggernaut newspapers—which are doing professional jobs at optimization.)

I’m very sympathetic to the dilemma newspapers are experiencing. I recall a time not long back when newspapers felt that internet yellow pages companies (”IYPs”) were as much of a threat as Google (see Local Media Face Growing Threat from Local Search Competitors Like Google, Overture and Yellow Pages, New AIM Group Study Reports and Search Engines Make Local Landgrabs, Leave Newspapers Out In the Cold), back when yellow pages companies had a considerable head-start over newspapers in online engagement and ad sales.

Since I used to work at an IYP, I also experienced firsthand what it was like to see a veteran print industry work to evolve to fit in the changing landscape while still being influenced strongly by legacy technologies. Technology wasn’t the only issue: organizational resistance toward seeing where things were headed, or even relatively insightful observations that there might be a risk in not engaging more aggressively also held the IYPs back. Since newspapers perceived the threat beginning such a long time ago, it’s disheartening to see that as a whole they have struggled to develop an effective adaptation for online—particularly the smaller, local market papers.

So, what’s to be done?

While there are a great many areas where online newspaper sites might improve and increase revenue prospects, one of the greatest untapped potentials on newspaper sites in my opinion is the news archive section. Even among poorly optimized newspaper sites, some articles may vanish into a walled-garden archive section at some point, going dark for search engines. Combined with very poor on-site search utilities, it’s as though these articles don’t exist at all for consumers.

I can’t count how many different newspaper sites I’ve visited where I’ve searched for articles which I knew existed, yet the on-site search engines could not locate them. In some cases, the “live” sites had search engines separate from archive search, yet offered no explanation to users as to which should be used and in what cases. Do articles pass into archive after one year? Two? Three? Why can’t the on-site search show them, regardless? In many other cases I’ve found articles by searching in Google, but the article is no longer available when I click through to the newspaper site, and searching within the site fails to reveal it. Did the article “expire” and pass into the archive graveyard or something? No messaging on the resulting error pages reveals this, nor suggests viable means for locating the article.

Newspaper folks: this is your main product! It’s all well and good to try to keep Google from making everything free and putting you out of business, but at this point there’s an even greater danger in locking away your content to the point where online consumers cannot even find—if a searcher doesn’t even know it exists, it’s certain they won’t be engaging with your site to try to obtain it, regardless of whether it’s provided “free” in return for ad impressions, in exchange for “free registration” or provided in return for some subscription fee.

How many articles are locked away in these old archives?!? It surely varies from newspaper to newspaper, but the potential numbers are staggering. While clicks on pay-per-click ads on newspaper sites may add up slowly, there’s no doubt in my mind that if newspapers dramatically expanded the content they have available to search engines, the clicks and associated revenue would increase. These newspapers must not realize the potential they’re sitting upon!

I acknowledge that current news is going to be the more popular content on newspaper sites, but there’s likely at least half as much traffic potential in the legacy content under the theory of the long tail. According to that theory when applied to newspaper website traffic, yes, there’s far more visits per contemporary news story than past ones, but the cumulative traffic from thousands and thousands of past news stories can equal or dwarf the traffic from the more popular stuff.

So, how should news archives be optimized for search?

Here are a few tips to get you started:

Optimize titles & headlines. This is one area where newspapers should utterly dominate! Reporters and editors often write beautiful article headlines which succinctly describe the topic and grab readers’ attention. But, the headline prose is squandered on some newspaper sites which either repeat the newspaper’s name for the TITLE text of all pages, or cram it up with the date, newspaper name and other “branding” messaging before the article title.

This amounts to almost criminal misuse of the title tag. The title is often the link text that’s displayed in search engine results when pages on your site are found to match the search term, and it’s displayed at the top of the browser window when a user visits the page. Also, in HTML there is a particular tag called the “heading” which is intended for just what it sounds like—used as way of identifying the heading and subheadings on a page, and it should be used when displaying article headlines. There are six different heading tags available (each uses different font attributes to add or decrease emphasis), but the main one you need to know is the <h1>, which is perfect for use in displaying an article’s headline on the page. The article headline should also appear at the beginning, not end, of title tags, and be displayed in H1 tags on the page for best usability and search engine friendliness.

Link to all of your content. To this day, search engines still rely heavily on links to pages to discover and index content. For good usability and crawability, I recommend designing a hierarchy of pages on your site so that users may click from the homepage to a page which provides a short list of top level links (such as links by dates or category of type of news story). Those top level links can link down to subcategory pages which link down further to each article ever published by your newspaper. Such a hierarchy of links is mainly for human site users to navigate down into all of your content, but it also helps the search engines understand the site structure, apply relative priority weighting of pages, and also to semantically categorize content.

Create and actively maintain sitemaps. Although the category pages I mentioned above are often loosely referred to as sitemaps, “official” sitemaps files (those created using a formal standard acknowledged by all of the major search engines) are lists of links to your pages that search engines use find all of your content. These should be used in conjunction with the hierarchy of links provided for human users. The sitemaps help insure that the search engines can find all of your pages.

Focus on creating evergreen URLs. Search engines tend to respect pages that have been around for a long while, and frequently changing the physical location of pages confuses both users and search engines alike. So, try to design article URLs which remain stable when an article is pubbed all the way through to when it’s archived for the long term. People also tend to link to articles, which helps search engines to decide how popular a page is. If you change the URL, then the link “votes” for popularity will no longer point to your article.

If you must change URLs, use 301 redirects. If your system is full of legacy processes which require you to change article page URLs once a current news article is moved off into the archive, then at least redirect the original URL to the final location instead of just delivering up an error page. Most users who click through won’t stop to poke around to try to find where something was shifted-to—they’ll just abandon your site to try to find info elsewhere. And, that redirection command should be a 301 “permanent” redirection in order to insure the search engines apply the original URL’s popularity weighting to the new URL.

Offer a “first click free” option. For those newspapers which require registration or subscription prior to showing archive content, read up on Google’s “First Click Free” program. This process allows Google to crawl your site content and index it, and a person who clicks through from Google can view and read the first page for free, but you can then require payment or registration for subsequent pages.

Create a subscription designation with Google. If you set it up with them, Google will allow you to have content crawled, but when users click through they must pay or register to see any of the article. This is less-preferred by Google since it’s a less satisfactory user-experience. If going this route, I’d recommend displaying a good-sized chunk of the article to users that click through, as a preview. In that way, they’ll feel a little less disappointed, and may be drawn in further to pay a subscription to see more.

Improve on-site search Just as a usability matter, consider using Google’s site search if your internal site search doesn’t work well. Once the pages have been optimized as I’ve outlined above, you could implement Google site search and perhaps improve your site’s overall usability.

There are certainly many other areas for optimization for newspapers, and this is not an exhaustive list of SEO improvements which could be done for articles. Simply exposing hidden and non-indexed archives would be a good start.

Google is trying to find additional ways in which to help the newspaper industry, such as its plan to roll out a micropayment system within a year. Micropayment is FAR better, in my opinion, than attempting to require someone to purchase a month-long or yearly subscription when they might want to access only a single article. I’d also suggest improving classified sections or partnering with many news sites for a multi-site subscription.

The local newspapers have lost a lot of marketshare to online news sites and aggregators, but evolving to improve their popularity and traffic in the internet economy could help them to take back marketshare and increase revenue. SEO helps with promotion and expansion of audience. If you’re a local newspaper in need of increased business, seriously consider beefing up your search engine optimization game.

Over the past year we have worked with a number of organizations that have chosen to relocate their sites from an existing domain to a new domain. One of the questions that always comes up early in the process is “how much traffic are we going to lose?” It is an excellent question and not an easy one to answer, but in today’s column I am going to explore that exact question.

Here are some of the types of changes that can have an impact on traffic or rankings.

Domain change. Any change in the domain, such as a move from http://www.old-domain.com to http://www.new-domain.com. The most common reason for doing this is a branding change of some sort. An existing business may be changing its branding, or one business entity may have been acquired by another one and the two sites are being merged.

Structural changes or URL changes. These are changes where the content that lives on a given URL on old-domain.com (such as about-us.html) gets moved to a different URL (such as about-us.php). URL changes can be “wholesale” (change nearly all or all of them), “heavy” (change a lot of them), “moderate” (change some of them), “light” (change only a few), or not done as all if you simply copy the exact site structure from one domain to another.

Structural changes often happen as a result of a change in the technology used to implement a site. For example, a business may have been using Cold Fusion as a content management system, and then switches to using ASP. The other major reason for structural changes is when wholesale content changes are made.

Content changes. Changes to the content on pages can happen without changing the URL structure of the site, by simply rewriting content on the pages, or something that causes structural changes to the site. As with URL changes, these can also be heavy, moderate, light, or not done at all.

Content changes may be made for many reasons. Perhaps the target audience has changed. Perhaps the basic positioning of the organization has changed. Another possible reason is to revamp the content as part of a wholesale expansion of the site.

Each of these things can happen independently. You can make content changes without changing the domain or the URLs. You can change the URLs without changing the domain or the content.

What are the true consequences?

You are going to lose traffic. That is a fact. Even if you only perform a domain change and preserve the exact same site structure and content, you will lose some traffic. In this simplest of scenarios you can minimize the amount of traffic loss by using 301 redirects from each URL on the old domain to point to the same URLs on the new domain, alerting the search engines that the new URLs are the important ones.

In principle, this simple domain change scenario sounds like one where there should be very little lost traffic. One factor to consider though is that of “trust.” Any time there is a domain change it may be reflective of an ownership change, even if the WhoIs info is not updated. For the search engines this raises the possibility that the new owner isn’t as trustworthy as the original owner.

Another factor concerns the 301 redirects themselves. In tests we have done at Stone Temple Consulting, we have seen evidence that they pass through the majority, but not all, of the link juice to the destination page. Sometimes there is a delay between the implementation of the redirect and when the search engines pass through the link juice, which can result in a significant drop in search engine traffic. In the medium to long term a simple domain change is usually not that costly (though there are exceptions). You may lose 20% to 40% of your traffic in the short term, and 10% to 20% in the medium to long term.

As you might imagine, the more complex the changes, the greater the potential negative consequences. For example, combining a domain change with URL changes will definitely be more costly. You have given the search engine more reasons to trust the site less, and your 301 redirect map just got more complicated. Assuming you completely restructure the site so all the URLs change, you can expect to see traffic loss of about 30% to 50% traffic loss in the near term, with gradual improvement on that over the longer term.

In our final scenario, if you change your domain, URL structure, and your content, you are asking for trouble. The big reason for the cost here is that the new content you create is not the content that people saw when they linked to your site in the past, even if it basically about the same subject matter. This probably results in the search engine significantly discounting the value of those links. Traffic loss in this scenario is likely to be 50% or more in both the short and long term.

To summarize:

Disclaimer! These numbers aren’t exact, and your mileage will vary. The actual impact of changes to your site will depend on many factors that are not possible to cover here. In addition, the chart assumes that you don’t do a lot of incremental link building to bolster rankings. However, savvy site owners rarely stand completely still.

How to mitigate the damage

Once you have made your changes, and assuming you have followed Google’s recommended best practices for doing so, the main damage control you can do is get new links to the site. In particular, if you can continue to get links at a pace similar to, or better than, what was happening before the move, this is a strong positive signal to the search engines that all is well.

Also make sure that you ask people who have linked to you in the past to update their links to go direct to the new site, bypass those pages where you’ve put 301 redirects in place. If a significant percentage of your past linkers do this it is also a very strong signal to the search engines that your site is still trustworthy in its new location.

The best remedy? Stop and think about the consequences of a move before committing. There will be a cost, and your business plan probably does not call for a dip in traffic to, and orders from, the web site.

The bottom line: Don’t make major changes to your site unless you really have to, and are willing to pay the price.

After last week’s “Does SEO = Spam” debate erupted, I had a number of follow-up emails where I explained privately more about why SEO has such a bad reputation in some quarters, as well as what SEO is and isn’t, from my perspective. I wanted to share some of that below.

Bad Advice Sucks: From SEOs Or Web Developers

I can completely understand the frustration people can feel when they hear SEO advice from someone positioning themselves as a “professional” that seems to be all about tricking search engines. As someone who has been preaching good, content-driven SEO since 1996, I also encounter people who get confused or offered bad SEO advice.

At the same time, I have many friends and colleagues in the print industry who struggle with web development “professionals” who think they know all they need to know about search engines. These professionals will refuse to implement 301 redirects rather than 302s, telling the SEOs that they somehow know better. And thus we get situations like when the International Herald Tribune lost pages it had in Google.

Time and again, you also discover some developer who created some all-Flash or all image web site. They get the basic idea (eventually) that search engines are, by and large, textual creatures. They want text in the way that radio wants sound and TV wants pictures. So they give the search engines some text in the wrong way, hiding it all with CSS code to make the text invisible by rending it in the page background color or pushing it off the visible screen entirely.

SEO Doesn’t Equal Spam

Publications do need good SEO. My concern about anti-SEO posts is that they often equate SEO with spam, as if they are the same thing. That’s like saying email marketing is the same as spam, or that all advertising is spam, or that all web development is the same as having badly designed web sites.

Yes, publications should have good writers. They should have good content. Trust me, a good SEO would love nothing more than to deal with a site that has outstanding content from the get-go.

SEOs Love Content But Need To Speak Louder About That Love

I realized that as part of this debate, as with similar debates I’ve engaged in over the years, that many outside the SEO world don’t understand that within it, plenty of good SEOs know that content is king. It’s understood. We know that’s the foundation, so we don’t focus on it any more than someone might focus that to be alive, you should breathe air.

But it’s a good reminder to everyone — content rules. Anything you do builds on top of that. And without it, you have a longer struggle. And without it, you might decide on shortcuts that don’t yield long-term gains.

The Spam Speaks Louder Than The SEO

There is indeed search spam that happens, as you’ll find with any other type of marketing activity. The worst, which I hate more than anything else, is automated link spam or comment drops. It adds no value to anyone. It creates major headaches for site owners. It causes people to assume that all SEO is like this (it is not).

In 2005, I tried to get the greater SEO industry to come out against this in a loud way. I had no particular luck. Last year, I wrote a long piece that called bullshit on the practice. In some keynote talks earlier this year, I also spoke that it shouldn’t even be a joking matter in some quarters. That to me, it’s becoming like hearing someone tell a racist joke. Yesterday, I also posted a personal look at how link spam has impacted my own wife’s web site.

Link and comment spamming won’t go away. It just won’t. It is unfortunately what we suffer from having open systems that allow people to submit anything without moderation and review, in a climate where occasionally, gathering links in this way can still work in the short term. But I’d love to see it go. I also don’t know any reputable SEO who is going to suggest to any reputable publication or web site that they should link spam the hell out of the web. If someone raises that as part of their SEO “plan,” publishers should run fast the opposite direction.

Related to this would be the creation of low-quality doorway page or microsites that have no other purpose that to be search engine fodder. That still goes on. I wrote about it in 2005, saying:

I come across search spam all the time — which to me is irrelevant content that’s overtly attempted to get a good ranking. I dislike it immensely when I hit this type of content, because I know exactly what the person has done to be misleading.

Yes, I’d like to see that go. I’d like to see people focus on building out really good content that works in the long term.

SEO Is More Than Web Pages

Just as SEO does not equal spam, it also does not equal just making sure your web pages are search engine friendly. SEO these days also covers things like video search, local search and blog search. Some of these aspects are well outside what a web developer would normally deal with. Even some web search aspects might not be familiar to web developers (such as how Google creates those small sitelinks below some listings).

Part of the bedrock of all this is keyword research. Plenty of people who do personal blogs will be vocal about how they rank for whatever they want or that they don’t care, they have no terms in mind that they are aiming for. That’s terrible advice for the majority of people out there.

Search engines are where people go to seek information. They express what they want using their own words. They’re having a conversation with that search engine, and part of the conversation back comes from the web sites that hear what they say and speak the same language. If they’re asking for “concrete security barriers,” but you’ve written your entire web site NOT using those words and only saying “revetment,” you’ve hurt your chance to speak to that person.

I don’t think web developers typically think of the keyword research aspect. They typically are not marketers. Heck, it’s hard enough to get writers to think of that aspect. But they should. A good SEO, aside from site architecture and inclusion issues, would also ensure that an organization is tapping into keyword research tools or has some awareness of the important of the words they choose.

That does not mean then stuffing all those words into a page. It doesn’t mean writing nonsensical articles that repeat the terms in a mishmash. It doesn’t even mean that you always say “concrete security barriers” instead of revetments on every page. But you do understand that you should use the terms searchers are using in an appropriate way.

SEO Is More Than Search Engines

Let me also revisit my definition of SEO:

SEO — search engine optimization — is the process of getting traffic from the “free,” “organic,” “editorial” or “natural” listings on search engines. All major search engines have such listings, where web pages, web sites and other content such as videos or local listings are shown and ranked based on what the search engine considers most relevant to users. Payment isn’t involved, as it is with paid search ads.

Anywhere people search, there are listings. SEO is understanding how these listings are generated and ensuring that you are well represented within them.

That’s why in the definition above, it’s important to note that “search engines” is broader than traditional search engines. For example, Urbanspoon is a popular mobile application to locate restaurants. You don’t even have type type in a keyword to show up there. Yet there are SEO aspects to how you are listed. If you’re a restaurant, part of your SEO efforts include understanding how Urbanspoon gets its data and the things you should be doing to appear well within them.

That understanding is more than what a typical web developer will do. No offense to the web developers out there. It’s just that keeping up on web development — the actual development of a web site — is a full-time job. Trying to also master the marketing of a web site on things that are not traditional search engines, that don’t even read your web pages in some cases, is probably more than a single person can do. That’s where a good SEO or a good marketing person can help. Together, they make a great team. They shouldn’t be enemies.

The Bad Spam Keeps The Debate Going

Why does this debate about SEO continue? Certainly there is a problem. The SEO industry has a terrible reputation. In part, it’s because the worst part of it makes the most noise. The people who most need work go out and cold-call, cold-email or whatever. The good professionals aren’t doing this. That’s because they’re often fully employed and can’t take on more work.

Also, anyone can call themselves an SEO, but we have no defined standards of what that is. So, you can get bad people doing bad work that generate a further poor reputation for the industry overall.

Unfortunately, the industry itself also has never been able to unify around standards. One reason is is because there are some people who disagree with the search engine standards that are issued. Google doesn’t like paid links, for example. Some SEOs simply think that for their industries, they have no choice but to buy them. Certainly you have plenty of “reputable” publications willing to sell them, who get nothing but a slap on the wrist from Google if they do so. No one has actively tried to pursue this further.

One thing I’ve suggested in the past is that perhaps there can be unification on deliverables. Many SEOs, regardless if they are “black hat” or “white hat,” likes the idea of someone who is just ripping off a client, not really doing any work, overhyping what they can really do or offering “guarantees” that when you look closely aren’t guaranteeing anything worthwhile.

The “Bad” Web Developers Impact Fewer People

Having said all this, there’s no standard among web developers. Anyone can simply say they are a web developer, but they don’t have to necessarily follow some accepted practices. You’ve got web developers who do sloppy work, who can create a mess of a web site. And I get out-of-the-blue annoying cold call emails from them, as well. Just fewer than the SEO pitches. Also, bad web developers only tend to impact their clients. Bad SEOs — people doing the comment spam — impact many others across the web, just as email spammers do.

Developers Have A Responsibility, Too

Despite all the spam that gives SEO a bad name, I think the design and developer community adds to the problem by often equating SEO with only the bad, without realizing the good out there.

Good SEO is often good site design. Good SEO can go hand-in-hand with good web development. But in my years of dealing with developers, it seems a second or third priority thing to some of them.

The top aspect is just build the site, test it with the major browsers, maybe test it with some users. Far too often, no one considers the impact those long, dynamic URLs may have on search engines. Or that Flash remains largely invisible. If SEO is considered, often it just seems dismissed as the crap it is perceived to be, rather than the assistance it can provide.

Alternatively, SEO is dismissed with a “I shouldn’t have to build my site to please search engines” attitude. That’s like saying you shouldn’t have to adjust your all-visual TV ad to air on radio. Except, unlike that example, SEO considerations are far less radical and help users, too.

Ironically, web developers and designers also fail to recognize how much work has been done by SEOs that improve their lives. It hasn’t been developers who’ve lobbied for Google and other search engines to better understand dynamic URLs, for example. That’s been the SEO community. Similarly, the serious issues with duplicate content? Years, and I do mean years, of lobbying by the SEO community helped bring forth the canonical tag. Meanwhile, if people are less able to “hijack” your listings in Google, you can thank SEOs who kept hammering Google on this issue until they improved things.

Can’t We All Get Along?

Derek Powazek, who kicked off the latest debate last week, has done a fresh post apologizing for grouping the good work with the bad:

To the people out there doing good work for real clients under the auspices of SEO: I’m sorry. I lumped you in with the bastards because I thought of what you did as web development, not SEO. I cast too wide a net and caught some good fish in there with the bad. I apologize.

That’s appreciated. And his advice to those in SEO should be also heard:

If there is going to be such a thing as “good SEO,” then the good guys need to fix their industry – put a stop to the evil practices or find some way to distance themselves from the evildoers. The way to silence critics is not to attack the critic, but to change the target of the criticism (especially if the criticism is justified).

I agree. But all I can say is that’s there’s no Team SEO that can swoop in and wipe out the spam, any more than Derek can police web developers who run ads in the Wall St. Journal promising to do both web sites & SEO for the low, low price of $695. Or the TV ad I just saw this weekend from Intuit, suggesting that for only $5 per month, I can have a web site and also get found on search engines.

Bargain. For only $60 per month I can have a site & rank on search engines? Clearly both web developers and SEOs are big rip-off machines, right? All you need is a template.

The good SEOs can’t stop the spam, but they can distance themselves from it. And they do so, time and again responding to posts like those from Derek and others over the years, to speak up and say SEO is not spam. They speak, they preach, they educate. I guess they’ll have to keep speaking, preaching and educating even louder.

But as they do it, I hope they do it with patience and care. Avoid personal attacks. Take the high road.

I’m often asked why I don’t give up. The reason is that people do listen. You can have conversations and attitudes can change. Make more good SEO visible, and maybe the spam won’t be the main thing that seems to speak for the industry.

Oh, that this weren’t true. Last week, Yahoo made news by disclosing that it had quietly dropped support for the meta keywords tag. As a long time hater of that tag and the insane questions it has produced, I was thrilled! But today, I see conclusively that Yahoo still supports the tag.

The test was simple. I placed a unique word in the meta keywords tag on the home page of Search Engine Land. This word — xcvteuflsowkldlslkslklsk — generated no results on Yahoo when I looked earlier this week. Today, when I searched, it brought back the Search Engine Land home page. Thus, Yahoo indeed indexes the content of that tag. (And to be clear, I looked before writing this article. In short order, this article itself, along with others, will appear because they’ll make use of that word).

During the session last week at SMX East, when Yahoo said it no longer supported this tag, several in the audience said they didn’t believe it. I was kind of struck. You’ve got a search representative flat-out saying they don’t do something, but no one wants to believe them? How things have changed. Sure, I can see distrust on some controversial issues (such as whether Google really does not count nofollowed links out of Wikipedia). But why would Yahoo lie about something like meta keywords support?

To be clear, I don’t think Yahoo was deliberately lying. The representative was probably confused in some way. Similarly over at Bing, despite them NOT supporting the tag (it’s not mentioned here) and never having done so since they launched their own search technology, they recently blogged much advice about using the tag.

As I commented about this:

That reads like someone got a copy of really old SEO advice and decided to put it out there regardless of what Bing actually does. I mean, my head hurts, but not everyone cared about commas or not. And no one had this 874 character limit. I mean, if you went over, it was no big deal. And the don’t repeat more than 4 times? According to what. Microsoft never, ever had its own guidelines like this.”

It’s a good reminder to the search reps. In many ways, you occupy god-like status on issues relating to SEO. Everything you write, everything you say will be fully believed by some. And if you’re not correct, you’ll confuse people and cause others to lose faith in you. If you don’t know, don’t say — or qualify: “I’m not sure” or “I’ll check on that.”

Postscript: Yahoo’s sent me this:

What changed with Yahoo’s ranking algorithms is that while we still index the meta keyword tag, the ranking importance given to meta keyword tags receives the lowest ranking signal in our system.

Words that appear in any other part of documents, including the body, title, description, anchor text etc., will take priority in ranking the document – the re-occurrence of these words in the meta keyword tag will not help in boosting the signal for these words.  Therefore, keyword stuffing in the keyword tag will not help a page’s recall or ranking, it will actually have less effect than introducing those same words in the body of the document, or any other section.

However, when no other ranking signal is present, unique words that only appear in the meta keyword tag section of documents can still be used to recall these documents.

I guess Derek Powazek isn’t done with attacking SEO. Now he’s published an SEO FAQ page which, sorry, really doesn’t provide much FAQ about SEO. So what the heck. I’ll deconstruct it. Be sure to also read my previous post, An Open Letter To Derek Powazek On The Value Of SEO.

What’s SEO?

First, any good FAQ ought to define what the subject is, right? So what’s SEO? Turns out, we have an SEO page here on Search Engine Land that explains this succinctly.

SEO — search engine optimization — is the process of getting traffic from the “free,” “organic,” “editorial” or “natural” listings on search engines. All major search engines have such listings, where web pages, web sites and other content such as videos or local listings are shown and ranked based on what the search engine considers most relevant to users. Payment isn’t involved, as it is with paid search ads.

You can read more on our page, along with some links to past coverage, a guide from Google and additional information

Notice in the definition above, I’ve mentioned things like videos or local listings. Search engines, and SEO today, aren’t just about web pages. That’s why when Derek rants that SEO is just some web developer’s job, he exposes a real lack of knowledge of the current state of SEO itself. In turn, frankly, he’s not giving out good advice.

SEOs Are All Scumbags

You know, I went to the gym today. Exercised with a personal trainer. All that trainer did was do stuff with me that should be common sense if you read a book, looked at the instructions on the machine and so on. Why isn’t he some scumbag that’s ripping me off? He’s a professional. He knows exercise intimately, in a way that I do not. It is his job to be an expert in this area. I want that expertise, someone who is totally dedicated to that area, not someone who dabbles in it.

SEO is a profession. Companies from MTV to the New York Times to the Wall St. Journal to Yahoo, to name only a few, employ full-time people who are responsible for SEO (and they aren’t scumbags, either). These companies have some of the best content in the world. And yet, they can still have major issues in how their sites are built or written or constructed that prevent them from doing well in Google or other search engines.

These SEOs, by the way, struggle with web developers who “think” they know SEO but don’t. Web developers who think that despite what an SEO tells them, a 302 redirect is the way to go. And thus the International Herald Tribune loses thousands of links because who wants to trust the scummy in-house SEO, right? I’ve got story after story of web developers and designers who think they know SEO but don’t, who cause major problems for web sites, and yet NO ONE ever writes a blog post blasting them.

There are bad SEOs. There are scummy people who there who spam blogs, who try to go black hat against search algorithms, who figure all’s fair in the fight for traffic. There are also bad people in any industry out there. You don’t count out an entire industry with helpful people because of your outdated ideas of how things work on the web. Especially if only deal with a particular type of web site (Derek doesn’t, for instance, seem to deal much with shopping sites, as best I can tell).

Deconstructing Derek’s FAQ

With that setup, let me go through the “FAQ” points Derek’s put out there and comment on them:

I publish a magazine and I know a lot of magazine publishers. And they are forking over embarrassing sums of money to charlatans who say they can raise their search engine rankings. These magazines can barely pay their writers. That’s wrong and it has to stop.

If you’re a company that’s about to pay some SEO expert, please, I beg you, take that money and hand it to a talented writer or competent web developer instead. It’ll be much better spent.

If you’re a company that hasn’t reviewed your site for SEO aspects, I beg you, consider this. Unless you like the idea of potentially wasting money, it pays to ensure you’ve done the basic things out there that will give you traffic for free. Maybe everything’s all perfect already, that your web developer who also knows some things about SEO has it all down. But it’s cheap insurance for many sites with a budget to ensure they’ve implemented SEO properly. Or, you can buy AdWords and pay by the click for that traffic from Google.

As for charlatans out there, yes, there are some. Just like there are some bad writers, bad designers and bad web developers. Ask for references. That’s not rocket science.

But I use SEO for good. Then you’re called a Web Developer. Good web development includes using proper formatting (like putting headlines in H tags) and understanding how the web works, search engines included. Valid code also has the side-effect of making your pages more accessible for your users, which is the point. Making your pages more accessible to robots is for robots.

Hey, I can fire up WordPress using the Thesis theme and make a nice site. Am I a web developer now? No need to pay some web developer hundreds of dollars per hour. I’ll do it for you as well.

Web development is not SEO. Good web developers will understand the fundamentals of SEO, in terms of good site architecture, crawlability and so on. But they probably won’t be ensuring that authors of a site are tapping into keyword research tools to ensure that when they write an article, they’re using terms that an audience seeking that article might use.

More important, few of them are dialed into how to handle giving Google and others a shopping feed. Or a feed of real estate listings. Or the completely separate ranking aspects that impact YouTube (the world’s second most popular search engine). Are they putting out a full-feed that Google Blog Search prefers? Are they checking that the URL shortener you use on Twitter spits back a 301 rather than a 302 redirect or worse, frames stuff up via a 200 code?

This isn’t 1999. This isn’t put stuff in a H1 tag and you’re good. This isn’t a world where we lack important tools such as the ability to feed sitemaps or even today, a way to see pages exactly as Google sees them. SEO encompasses any type of search engine dealing with any type of content.

[Insert irate defense of SEO here.] You sell SEO, right?

Yes, some of the people who are defending SEO also either perform it or are related to the industry. And some of those same people will readily admit that the industry has a terrible reputation. But then again, they actually KNOW SEO. They’re responding because unlike people like Derek or others who launch rant posts like his every six months or so, they deal with confused people all the time. They pick up the messes that web developer make. They solve the real problems out there that just being “real” won’t help. The don’t deserve to be written off. They actually deserved to be listened to, because they have lot of valid things to say.

If you’re so smart, prove it. I’m not that smart. But I can say that I am Google’s third result for “Derek” (I was number one until Wikipedia came around). And after less than 24 hours, my post about SEO is the ninth Google result for “SEO”.

Well, if Derek’s SEO strategy was to find people and convert them into sales for those seeking “Derek,” congrats. But actually, part of SEO is understanding what a target audience is after, in terms of your content. And I don’t care who are. Unless you’re doing a personal blog with absolutely no particular audience in mind, you do have a target audience. You’d better understand how that audience might be seeking your content. Otherwise, you actually might not get found. Good SEO starts with good keyword research or at least an awareness of it.

As for Derek ranking of SEO, welcome to the world of “query deserved freshness,” where Google will reward fresh posts with a relevancy bump. Does your web developer understand that? Because if they don’t, then when their boss starts complaining in a week how your great ranking just went away, they might start messing with your site to regain it — and make things worse.

How did I accomplish these magic SEO feats? Exactly how I said I did: Make something great. Tell people about it. Do it again. I’ve been doing that (or at least trying to) since 1995.

In this case, I wrote a passionate post. I posted it to Twitter and Facebook. And I sent one email to a friend about it. That’s it!

Sure, Derek is passionate about an industry that has a lot of haters. Guess what. I’ve been writing about SEO since 1996, telling people the good ways to do it and why it helps. And you know, it HAS helped lots of people.

If SEO is just all snakeoil, why am I still here? Why is SEO still here? Derek is hardly the first to have a rant. I’m easily into double-digits of rants like his I’ve read over the year.

The answer is that it ISN’T all snakeoil. SEO really is and can be useful. It’s just a pity that more people who do get helped by it who aren’t SEOs themselves don’t speak up for it. But here’s another tip. If your company does well with SEO, you usually don’t want to talk about it — because you’re hoping your competitor is listening to people like Derek who say SEO is all bullshit.

SEO is needed because of bad web design. Wouldn’t it be better to make competent websites in the first place?

Yes, it would. And when the miracle happens when all web developers are doing that, please let us know. In the meantime, perhaps instead of antagonism, we can talk about ways that a good web developer can identify a good SEO to work with? And also understand that even then, SEO extends beyond web pages these days.

It’s easy for you – you’re an expert. I’m using WordPress (which is free software) and the DePo Skinny theme (which I designed and released for free). You can download both of those and have a site just like this in a few minutes. No need to pay anyone for SEO. Then comes the hard part: Spend over a decade making things. If you do that, you’ll be an expert, too.

So just be an expert in selling ranch fencing, and you’re good. Or one on hazardous waste disposal in your local community, and you’ll rock. Or toss up a blog about your local locksmith site, and you’ll be cooking along. Or not (see Searching For Small Businesses, Coming Up Frustrated).

Passion is great. Wonderful content indeed rocks. And even “boring” businesses can tap into viral popularity. But you also have some boring SEO stuff you probably want to attend to, as well.

[Insert personal attack here.] I may have tarred a so-called “industry” but I didn’t attack anyone personally.

Really? Let’s review some statements:

Search Engine Optimization is not a legitimate form of marketing.

If someone charges you for SEO, you have been conned.

A new breed of con man was born, the Search Engine Optimizer.

SEO cockroaches employ botnets, third-world labor, and zombie computers to blanket the web with link spam.

SEO bastards are behind worms that attack blog services like Blogger, WordPress, and Movable Type.

It’s a game, and every link is a score for the SEO jerkwads and their disreputable clients.

Yeah, that’s a personal attack. Derek is calling each and every SEO out there someone who is not legitimate, someone who cons, someone who apparently runs botnets and worms and only has disreputable clients.

None of that’s qualified. He points it at each and every person who specializes in SEO. Part of me is amazed more people haven’t emerged to call him out on this type of hateful, unfair and inaccurate attack. But part of me figures these days, plenty of big voices hear SEO and either don’t want to speak up for them or based on their experiences as blogging rockstars think that everything that works for them must work that way for everyone else.

This may be obvious to you, but it’s not to everyone. You’re right. I regret saying that this stuff was obvious without explaining what I meant. Here’s what I meant: Good SEO techniques are just good web development techniques. They should be obvious to anyone who makes websites for a living. If they’re not obvious to you, and you make websites, you need to get informed. If you’re a client, make sure you hire an informed web developer.

So I heard Chris Brogan speak recently, and he joked that his book Trust Agents (cowritten with Julien Smith) was just full of common sense. Which is true. But what’s common sense to someone who knows their field is uncommon wisdom to those who don’t. Some SEO techniques are also good web development ones. A good web developer should know them or should educate themselves. But plenty do not. Or plenty assume they know it all when they do not.

CLIENTS: If someone approaches you about optimizing your search engine placement, they’re running a scam. Ignore them. If your site isn’t showing up in Google, fire whoever is making your web pages and hire someone better. Sign up for social media services (Twitter, Facebook, etc) and participate there. Pay for quality writers and designers – that’s what will actually raise your ranking in the long term.

Agreed, if someone approaches you out of the blue — you get that email offering to do some analysis, etc. Ignore it. Might not be a scam, but it is generally not a sign of a company that’s in demand because of a satisfied client basis. As for firing your web page builder, um — maybe. Or if you like what they’ve done with the site, and SEO might be able to work with them and allow you the best of both worlds.

WEB DESIGNERS: Learn to code your own pages. If you can’t, hire someone who can, and listen to them when they tell you why putting all that text in an image is a bad idea.

Those people who can, by the way, are called SEOs. You know, those scumbags you shouldn’t be listening to.

WEB DEVELOPERS: Educate your designers about proper web development. Educate your clients about how the web works. Follow Google’s advice. Read A List Apart. Writing good code won’t just help your Google rank, it’ll make certain your site is accessible to screen readers, mobile devices, and all the browsers out there.

Well, I don’t regularly read A List Apart, so I can’t comment on the quality of the SEO advice over there. But yeah, read Google’s. In particular, read their SEO starter guide here.

You might take in a conference or two. At our SMX East conference last week, we had an entire track devoted to technical SEO and developer issues. Want more? Jane & Robot started last year is devoted to developer & SEO issues, and they have events.

You can check out topics here on Search Engine Land, such as our 100% Organic column and our Google SEO, Microsoft’s Bing SEO, Yahoo SEO and How To SEO sections.

SEO SPECIALISTS: If all you do is SEO, you need to expand. Hire a visual designer and some kickass coders and become a real web agency. Start making sites good from the get-go instead of cleaning up other people’s messes. Besides, if all you do is SEO, your days are numbered. Social media is rapidly becoming much more important than Google. (Number one referrer to my site this week? Twitter.)

Again, this isn’t 1999. There are plenty of agencies that offer SEO services. Some of these were pure SEO companies that grew. It’s also no newsflash to many good SEOs that social media has grown in importance. In fact, the real news seems to be that Derek doesn’t realize they already know this.

GOOGLE: Would you look at all the crazy you’ve created? You could fix this by making your engine more inclusive of websites with common mistakes, introducing some randomness to the order of results (why should everyone’s results be in the same order?), and unforgiving punishing the businesses that have sprung up to exploit your popularity. Get on it.

Yes, they do. That’s one reason why Derek’s ranking for SEO at the moment. It’s also the reason that plenty of people like him can say you don’t need to do SEO. Because for lots of them, especially if they have promient blogs, Google does do the right thing.

But it’s not perfect, and you can help it. That’s why over the years, it has rolled out a ton of tools at Google Webmaster Central that any good SEO will know about.

Complaining that Google should just “fix” everything also reminds me of web developers who have long assumed that problems with search engines and Flash would just eventually go away, so they didn’t need to do anything. I’ve heard that for over 10 years. Despite advances, the basic advice remains the same. Try not to lock your content up in Flash.

When developers will test their pages to make sure they work in Internet Explorer, Firefox, Safari and Chrome, not testing against the biggest browser of them all — Google — is foolishness. What Derek’s written above is the same as complaining that Internet Explorer has some bug that’s screwing up his web pages, and rather than fix it, he thinks IE should solve the problem. Sure, IE should — but in the meantime, you workaround it.

Postscript: Let me make clear my intent in this piece isn’t to attack Derek personally. I have issues with many of his statements. Since these statements seemed based on his personal experience, it’s difficult to avoid addressing him on this by name. I’ve read through this piece again, and I really don’t see a personal attack him Derek. On Twitter, in reference to his original post, I did say things like:

yes, let’s make heroes out of people who rant ignorance & harm others who do good work. hurray! (here)

& sadly plenty of SEOs provide good non-sleezy, useful, legit, google-backed services but get tarred with usual ignorant brush (here)

see my tweets earlier today. it’s just the latest in years of ignorant slamming of SEO but (here)

I didn’t mean a personal attack by calling what Derek wrote a rant. That’s just a fact. It was. He called it the same himself. Heck, I rant all the time. There’s nothing wrong with a good rant, though I prefer one that’s also thoughtful and backed up by facts.

That’s where the ignorant part is. I don’t think Derek is ignorant or stupid in general. But I do think much of what he wrote seems to be ignorant of the current state of SEO. Ignorant as in not aware. A better choice of word, which might not have seemed perhaps personal, would have been uninformed.

Anyway, hopefully Derek and I will talk more about this tomorrow rather than playing dueling posts. My intent was never to attack him as an individual, but I certainly wanted to raise some awareness and balance to the issues he put out there.

Derek Powazek launched an attack on SEO yesterday that really said nothing that others haven’t ranted about before. I’ve responded to many of these attacks over the years in hopes of educating people about mistaken assumptions. I’ve largely given up. But I figured this time I’d give it another go with some personal illustrations I’ve encountered recently.

In particular, rather than do an article to counterbalance Derek’s post, I started to write him a private email. But as I composed that, I felt it might better illustrate to everyone why SEO is indeed a legitimate form of marketing and those who provide the service are not all “scammers” who are out to “con” you.

So Derek, I saw your rant, and it was disappointing. Your post was based on “14 years of hits and misses.” Well, my response come from my own 14 years of covering search engines. Of having answered feedback from hundreds of people. Of having talked with hundreds of people personally. Of understanding that the “you just build it; you just put it out there” approach to search engines, sadly, doesn’t always cut it.

Let me be clear. I totally agree with your core advice. Build a site for visitors. Have great content. These are the keys to success, not just with SEO but with anything you want to do. In fact, we just had an article on our site here reinforcing this.

Still, sometimes people have problems. And the stuff that you think isn’t rocket science — that anyone knows — is indeed a mystery to others. They want help, and sometimes they can’t find that web developer who also understands SEO issues. In the same way, you sometimes don’t find web developers who are also designers. Or designers who understand conversion issues. Or conversion experts who understand web development.

Let me tell you some stories.

A Mother Who Sells Homes

Two weeks ago, our local elementary school had a mixer for parents. I was talking with one woman who asked what I did for work. “I write about search engines,” I told her. That led her to asking if I know about how people get found on Google. Yep. So she started asking about her local real estate site, and how she might market it, things that she might do.

That’s a loaded question. See, for you, she just needs to believe in the real estate she’s selling, then tell her friends with personal notes, get out on Twitter, find places where her community congregates and “be real.”

But to really be real, let’s remember that she’s selling real estate in one of the most competitive areas of the country, Newport Beach, California. Her friends aren’t all going to buy homes she’s listing. Her “community” congregates on Google and does things like type in “newport beach homes for sale.”

To succeed in attracting that audience, she should have a great site and great content — agreed. But does she have individual listings? Then she probably needs to kick them out into Google Base, in order to fully be listed in Google. Does your mythical web developer deal with Google Base much? And where’s her web site now? Is she running it off Blogger? Using her own domain? These have impacts on how both the search engines may see her as well as how she’s perceived.

Does she have a blog in addition to a main site? That has an impact. Has she considered some unusual, creative ways to create content around real estate in her area, perhaps some catchy link bait, which may pull in the links she needs to rank better (which, by the way, is a recommended Google practice).

Does she have a local office? If so, has she claimed her listing in Google Local? If so, has she updated her title to reflect that perhaps she has “newport beach homes for sale?”

This is all SEO. It’s not your father’s (or mother’s) SEO that you rant about, the keyword stuffing, the link drops (none of which is best practices SEO anyway). But make no mistake, it’s SEO.

SEO & Baseball Practice

Later that same week, I took my son to baseball practice. I also took my computer, so that I could finish up work on an article and have the weekend clear when he was done.

Another father came over to me, asking what I do. “I write about search engines….” Which as before, led to the questions about how his company might show up on Google. Did I know much about that?

Sure. And since I had my mobile broadband card with me, I fired up his web site. Very sad. Same page title on every page. No keyword research employed, to think how people might be seeking out the industrial shipping cases they sell. Long, dynamic URLs that might pose indexing issues.

Where do you start with someone like this? In your world, his company should just “be real.” He sells industrial shipping cases. How real do you want to be about that?

His “community” are the people who realize they need a case like he sells and go online to places like Google and start searching for the products. And his pages are NEVER going to show up, because there’s nothing unique about his site and he had basic SEO errors that haven’t been dealt with.

If all he did was change things so that his page titles were different, he’d pull traffic. I know, I know — that’s so obvious. But it is NOT to him. His job isn’t to do web development. He’s not you or I with 14 years of having learned all this stuff along the way. He actually deals with things like ordering the products, overseeing workers and doing an array of offline marketing.

Meanwhile, his web developer clearly did NOT have any SEO thoughts in mind when building the site. That’s all too common. If I wanted to be snarky, I could do an entire post on why web developers are a waste of time and you should just employ SEOs who can also build web sites. But the reality is that a good SEO (and there are some) working to help direct the web developer could solve the site’s problems quickly. It’s called teamwork, and it’s awesome when it happens.

Mommy Bloggers & SEO

Over the weekend, I had a group of mommy bloggers over at my house. Not because I’m a mommy blogger but because my wife is a mommy blogger. The group was here because they’re all involved in a new project for mommy bloggers in Orange County.

My participation was to show up at the end of the meeting, not to say anything but just because I was coming home. I caught some of the closing conversation, and a question about Google came up. How was the search engine going to deal with their content, if these mommy bloggers put the same article on a publishers site as well as their own sites.

Answer? I don’t know. Google might decide to favor the publisher’s site not because it’s “more real” than the mommy bloggers but because it has built up more authority collectively over time. Certainly the mommy bloggers themselves are all “real” in what they write, how they put themselves out there to their communities and so on. Much depends on what they’re most concerned about.

If they had a key post that they absolutely wanted to be the primary source for, guess what? They’d better have some SEO savvy. They might need to tell the publisher to prevent the copy of their posts they provide from being spidered on the publisher’s site. Or if they allow it, then they might want to make use of the forthcoming canonical tag 2.0.

That’s not being “real.” That’s SEO. And that’s SEO that a good SEO will know — and many non-SEOs will not.

I’m glad, honestly glad, that you’re savvy enough to understand how good SEO can be incorporated into web development. I wish more web developers could do the same. But my experience has been that much good SEO gets overlooked. There are bad SEOs out there, who give the entire industry a bad name — just as there are bad bloggers, bad designers, bad cops, you name it. There are also excellent SEOs who work inside of companies as well as through agencies for hire. Don’t tarnish an entire industry that actually helps many, many people in ways I’m sure you would agree with.

For those who just can’t get enough of this subject, some of my past writings on the topic. They cover plenty of additional examples, plus the fact that Google itself recommends SEO:

• Dear Fox News: SEO Is Not Search Engine Scamming (Unless You’re Scamming Yourself): Aug. 2009
• Importance Of SEO Makes Front Page Of Los Angeles Times: March 2009
• A Bad Month For SEO’s Reputation: Feb. 2008
• From My Inbox: More Defense Of SEO: May 2007
• SEO: Real Skills That Can Protect Your Traffic: March 2007
• Why The SEO Folks Were Mad At You, Jason: Feb. 2007
• More Rounds In The “Is SEO Overrated” Debate: Jan. 2007
• Defending SEO, Yet Again!: Dec. 2006
• Yes Virginia, SEO Is Rocket Science – Defending Search Engine Optimization Once Again: Dec. 2006

See also the follow-up to this post, SEO FAQ That’s Not From The Land Of Unicorns.

Google Webmaster Tools has just launched a “labs” section, where you’ll find new features that may be early in the development cycle and not quite as robust as the rest of the tools. The features available so far are Fetch as Googlebot, which lets you see exactly what Googlebot is served when it requests a URL from your server and Malware Details, which shows you malicious code snippets from your site if it’s been flagged as containing malware.

Fetch as Googlebot

Of most interest to webmasters, SEOs, and web developers is likely the Fetch as Googlebot feature. You can specify any URL on your site and see the HTTP response (header and contents) that the server returns. Simply  indicate the URL and click the Fetch button. It may take a few moments for Googlebot to access the page and return the results, since it fetches the page in real time. (Refresh the page to see the progress.)

Click the Success link once it’s been processed to see the results.

How is this different from simply looking at the source code of the page?

• You see the HTTP header information at the top. This information is generally easily available through tools such as Live HTTP Headers, but isn’t contained in the source code itself (since that information is coming from the server, not the page).
• You can see if the server is returning any of the page information differently than the page has been coded.
• You can see if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (typically known as “cloaking“), you’ll see  what’s being conditionally served to Google.
• You can use the tool to test changes (particularly things like redirects) in real-time.

Note that this tool won’t necessarily show you the content that Google is able to extract from the page. If the page contains JavaScript, for instance, you’ll see the raw JavaScript code contained on the page, not the rendered view visible in the browser. Which, unfortunately means you can’t use this tool to determine if Google is able to access content contained in rich markup.

What’s this about cloaking?

This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re coming into a project late and aren’t sure what’s been previously done. It can also help uncover if your site has been hacked. Back in 2006, Googler Matt Cutts and I did a show on Webmaster Radio during which we talked about how in some cases, a hacker might add links to a site and then cloak those pages so that the site owner never sees them. Only Google does. At the time, Matt suggested using Google Translate (and choosing English to English) to see what Googlebot was being served, but this tool can now more easily serve that purpose. Matt confirmed this to me this morning: “The biggest use case is just debugging site issues. Of those, the biggest case will be hacked sites. Some attacks will hide content until search engines fetch the page (and some attackers add a noarchive tag so that the search result doesn’t have a “Cached” link), so a site could look clean to the website owner. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”

How do I test redirects?

If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. For instance, when I fetch www.searchengineland.com, I see that the redirect is correctly implemented as a 301 and points to searchengineland.com:

You can also use the tool to troubleshoot URLs listed in the Crawl Errors > Not Followed report. You can also test these URLs using something like Live HTTP Headers or by trying to access the URLs in a browser, but if neither of those methods uncover the problem, this tool can help determine that the issue is specific to Googlebot. You can also use this tool to verify that fixes you’ve made to redirect errors uncovered by the Not Followed report have really solved the problem.

(Note that the tool currently has a limit of 100kb per page. However, this is for the tool only and doesn’t apply to Googlebot’s normal crawl of the site. Google is monitoring feedback to see if many site owners find this size to be limiting.)

Malware details

The Google Online Security Blog has more information on the malware details tool. Previously, webmaster tools reported when the site was flagged has having malware and listed sample URLs. This new tool will also show samples of the malicious content, and in some cases, the underlying cause. This should help those site owners whose sites have been hacked to include malware find the problem and fix it. If your site does contain malware and you’ve fixed it, you can request a review to have the malware alert removed in search results.

And then there were none. Yahoo has long been the only major search engine that supported the meta keywords tag. However, the search engine revealed today that like the other majors, it no longer supports it.

The news came during the Ask The Search Engines session at SMX East in New York today. The search engines were all asked about their support of the tag. Moderator Danny Sullivan noted that only Yahoo provided support of the tag — prompting Cris Pierry, senior director of search at Yahoo, to announce that support actually had been ended unannounced “several” months ago.

Bing doesn’t support the tag. Google has never supported it and in fact clarified this again in a special post last month. See Google: Stop Suing Over The Meta Keywords Tag, We Don’t Use It for more about that.

NOTE: See our follow-up post, Sorry, Yahoo, You DO Index The Meta Keywords Tag.

Many site owners have wanted the recently introduced canonical tag to work across domains. Now their wishes will come true. Google announced cross-domain support will come by the end of the year.

The news came out during the Duplicate Content: The Search Engines Edition session at SMX East conference today. Google made the announcement in response to site owners again voicing that they wanted such a solution.

The existing canonical tag is only supported by Google, at the moment. Yahoo and Bing both said they’re studying support but think its likely they’ll add support by the end of the year. However, they’ll only support canonicalization across the same domain.

In a way, they’ll support Canonical Tag 1.0 by the end of the year, while Google will be supporting Canonical Tag 2.0, with cross-domain support.

We will have more information when more details come to us.

The Google AdSense for Feeds (aka FeedBurner) blog announced that all of their links will be 301 redirected, as opposed to some that are 301 redirected. In summary, when you use FeedBurner to create and host your RSS feed, FeedBurner creates special links that are used to send the reader to your web site. Some users selected to use a 302 redirected link, as opposed to a 301 redirected link, to obtain better tracking.

Google has made the decision to remove the 302 redirect option and make them all 301s. Why? The main reason is Google wants these URLs to become “more compatible with search engines that crawl feeds.” A 301 redirect is the best solution for a search engine to determine where a URL lives. 302 redirects historically have confused search engines, as well as webmasters. So to be “consistent with the way that content is distributed today,” Google has made this change.

If you run FeedBurrner for your web site feeds, you don’t have to make any changes. Google has already made the 301 change for you.