After last week’s “Does SEO = Spam” debate erupted, I had a number of follow-up emails where I explained privately more about why SEO has such a bad reputation in some quarters, as well as what SEO is and isn’t, from my perspective. I wanted to share some of that below.

Bad Advice Sucks: From SEOs Or Web Developers

I can completely understand the frustration people can feel when they hear SEO advice from someone positioning themselves as a “professional” that seems to be all about tricking search engines. As someone who has been preaching good, content-driven SEO since 1996, I also encounter people who get confused or offered bad SEO advice.

At the same time, I have many friends and colleagues in the print industry who struggle with web development “professionals” who think they know all they need to know about search engines. These professionals will refuse to implement 301 redirects rather than 302s, telling the SEOs that they somehow know better. And thus we get situations like when the International Herald Tribune lost pages it had in Google.

Time and again, you also discover some developer who created some all-Flash or all image web site. They get the basic idea (eventually) that search engines are, by and large, textual creatures. They want text in the way that radio wants sound and TV wants pictures. So they give the search engines some text in the wrong way, hiding it all with CSS code to make the text invisible by rending it in the page background color or pushing it off the visible screen entirely.

SEO Doesn’t Equal Spam

Publications do need good SEO. My concern about anti-SEO posts is that they often equate SEO with spam, as if they are the same thing. That’s like saying email marketing is the same as spam, or that all advertising is spam, or that all web development is the same as having badly designed web sites.

Yes, publications should have good writers. They should have good content. Trust me, a good SEO would love nothing more than to deal with a site that has outstanding content from the get-go.

SEOs Love Content But Need To Speak Louder About That Love

I realized that as part of this debate, as with similar debates I’ve engaged in over the years, that many outside the SEO world don’t understand that within it, plenty of good SEOs know that content is king. It’s understood. We know that’s the foundation, so we don’t focus on it any more than someone might focus that to be alive, you should breathe air.

But it’s a good reminder to everyone — content rules. Anything you do builds on top of that. And without it, you have a longer struggle. And without it, you might decide on shortcuts that don’t yield long-term gains.

The Spam Speaks Louder Than The SEO

There is indeed search spam that happens, as you’ll find with any other type of marketing activity. The worst, which I hate more than anything else, is automated link spam or comment drops. It adds no value to anyone. It creates major headaches for site owners. It causes people to assume that all SEO is like this (it is not).

In 2005, I tried to get the greater SEO industry to come out against this in a loud way. I had no particular luck. Last year, I wrote a long piece that called bullshit on the practice. In some keynote talks earlier this year, I also spoke that it shouldn’t even be a joking matter in some quarters. That to me, it’s becoming like hearing someone tell a racist joke. Yesterday, I also posted a personal look at how link spam has impacted my own wife’s web site.

Link and comment spamming won’t go away. It just won’t. It is unfortunately what we suffer from having open systems that allow people to submit anything without moderation and review, in a climate where occasionally, gathering links in this way can still work in the short term. But I’d love to see it go. I also don’t know any reputable SEO who is going to suggest to any reputable publication or web site that they should link spam the hell out of the web. If someone raises that as part of their SEO “plan,” publishers should run fast the opposite direction.

Related to this would be the creation of low-quality doorway page or microsites that have no other purpose that to be search engine fodder. That still goes on. I wrote about it in 2005, saying:

I come across search spam all the time — which to me is irrelevant content that’s overtly attempted to get a good ranking. I dislike it immensely when I hit this type of content, because I know exactly what the person has done to be misleading.

Yes, I’d like to see that go. I’d like to see people focus on building out really good content that works in the long term.

SEO Is More Than Web Pages

Just as SEO does not equal spam, it also does not equal just making sure your web pages are search engine friendly. SEO these days also covers things like video search, local search and blog search. Some of these aspects are well outside what a web developer would normally deal with. Even some web search aspects might not be familiar to web developers (such as how Google creates those small sitelinks below some listings).

Part of the bedrock of all this is keyword research. Plenty of people who do personal blogs will be vocal about how they rank for whatever they want or that they don’t care, they have no terms in mind that they are aiming for. That’s terrible advice for the majority of people out there.

Search engines are where people go to seek information. They express what they want using their own words. They’re having a conversation with that search engine, and part of the conversation back comes from the web sites that hear what they say and speak the same language. If they’re asking for “concrete security barriers,” but you’ve written your entire web site NOT using those words and only saying “revetment,” you’ve hurt your chance to speak to that person.

I don’t think web developers typically think of the keyword research aspect. They typically are not marketers. Heck, it’s hard enough to get writers to think of that aspect. But they should. A good SEO, aside from site architecture and inclusion issues, would also ensure that an organization is tapping into keyword research tools or has some awareness of the important of the words they choose.

That does not mean then stuffing all those words into a page. It doesn’t mean writing nonsensical articles that repeat the terms in a mishmash. It doesn’t even mean that you always say “concrete security barriers” instead of revetments on every page. But you do understand that you should use the terms searchers are using in an appropriate way.

SEO Is More Than Search Engines

Let me also revisit my definition of SEO:

SEO — search engine optimization — is the process of getting traffic from the “free,” “organic,” “editorial” or “natural” listings on search engines. All major search engines have such listings, where web pages, web sites and other content such as videos or local listings are shown and ranked based on what the search engine considers most relevant to users. Payment isn’t involved, as it is with paid search ads.

Anywhere people search, there are listings. SEO is understanding how these listings are generated and ensuring that you are well represented within them.

That’s why in the definition above, it’s important to note that “search engines” is broader than traditional search engines. For example, Urbanspoon is a popular mobile application to locate restaurants. You don’t even have type type in a keyword to show up there. Yet there are SEO aspects to how you are listed. If you’re a restaurant, part of your SEO efforts include understanding how Urbanspoon gets its data and the things you should be doing to appear well within them.

That understanding is more than what a typical web developer will do. No offense to the web developers out there. It’s just that keeping up on web development — the actual development of a web site — is a full-time job. Trying to also master the marketing of a web site on things that are not traditional search engines, that don’t even read your web pages in some cases, is probably more than a single person can do. That’s where a good SEO or a good marketing person can help. Together, they make a great team. They shouldn’t be enemies.

The Bad Spam Keeps The Debate Going

Why does this debate about SEO continue? Certainly there is a problem. The SEO industry has a terrible reputation. In part, it’s because the worst part of it makes the most noise. The people who most need work go out and cold-call, cold-email or whatever. The good professionals aren’t doing this. That’s because they’re often fully employed and can’t take on more work.

Also, anyone can call themselves an SEO, but we have no defined standards of what that is. So, you can get bad people doing bad work that generate a further poor reputation for the industry overall.

Unfortunately, the industry itself also has never been able to unify around standards. One reason is is because there are some people who disagree with the search engine standards that are issued. Google doesn’t like paid links, for example. Some SEOs simply think that for their industries, they have no choice but to buy them. Certainly you have plenty of “reputable” publications willing to sell them, who get nothing but a slap on the wrist from Google if they do so. No one has actively tried to pursue this further.

One thing I’ve suggested in the past is that perhaps there can be unification on deliverables. Many SEOs, regardless if they are “black hat” or “white hat,” likes the idea of someone who is just ripping off a client, not really doing any work, overhyping what they can really do or offering “guarantees” that when you look closely aren’t guaranteeing anything worthwhile.

The “Bad” Web Developers Impact Fewer People

Having said all this, there’s no standard among web developers. Anyone can simply say they are a web developer, but they don’t have to necessarily follow some accepted practices. You’ve got web developers who do sloppy work, who can create a mess of a web site. And I get out-of-the-blue annoying cold call emails from them, as well. Just fewer than the SEO pitches. Also, bad web developers only tend to impact their clients. Bad SEOs — people doing the comment spam — impact many others across the web, just as email spammers do.

Developers Have A Responsibility, Too

Despite all the spam that gives SEO a bad name, I think the design and developer community adds to the problem by often equating SEO with only the bad, without realizing the good out there.

Good SEO is often good site design. Good SEO can go hand-in-hand with good web development. But in my years of dealing with developers, it seems a second or third priority thing to some of them.

The top aspect is just build the site, test it with the major browsers, maybe test it with some users. Far too often, no one considers the impact those long, dynamic URLs may have on search engines. Or that Flash remains largely invisible. If SEO is considered, often it just seems dismissed as the crap it is perceived to be, rather than the assistance it can provide.

Alternatively, SEO is dismissed with a “I shouldn’t have to build my site to please search engines” attitude. That’s like saying you shouldn’t have to adjust your all-visual TV ad to air on radio. Except, unlike that example, SEO considerations are far less radical and help users, too.

Ironically, web developers and designers also fail to recognize how much work has been done by SEOs that improve their lives. It hasn’t been developers who’ve lobbied for Google and other search engines to better understand dynamic URLs, for example. That’s been the SEO community. Similarly, the serious issues with duplicate content? Years, and I do mean years, of lobbying by the SEO community helped bring forth the canonical tag. Meanwhile, if people are less able to “hijack” your listings in Google, you can thank SEOs who kept hammering Google on this issue until they improved things.

Can’t We All Get Along?

Derek Powazek, who kicked off the latest debate last week, has done a fresh post apologizing for grouping the good work with the bad:

To the people out there doing good work for real clients under the auspices of SEO: I’m sorry. I lumped you in with the bastards because I thought of what you did as web development, not SEO. I cast too wide a net and caught some good fish in there with the bad. I apologize.

That’s appreciated. And his advice to those in SEO should be also heard:

If there is going to be such a thing as “good SEO,” then the good guys need to fix their industry – put a stop to the evil practices or find some way to distance themselves from the evildoers. The way to silence critics is not to attack the critic, but to change the target of the criticism (especially if the criticism is justified).

I agree. But all I can say is that’s there’s no Team SEO that can swoop in and wipe out the spam, any more than Derek can police web developers who run ads in the Wall St. Journal promising to do both web sites & SEO for the low, low price of $695. Or the TV ad I just saw this weekend from Intuit, suggesting that for only $5 per month, I can have a web site and also get found on search engines.

Bargain. For only $60 per month I can have a site & rank on search engines? Clearly both web developers and SEOs are big rip-off machines, right? All you need is a template.

The good SEOs can’t stop the spam, but they can distance themselves from it. And they do so, time and again responding to posts like those from Derek and others over the years, to speak up and say SEO is not spam. They speak, they preach, they educate. I guess they’ll have to keep speaking, preaching and educating even louder.

But as they do it, I hope they do it with patience and care. Avoid personal attacks. Take the high road.

I’m often asked why I don’t give up. The reason is that people do listen. You can have conversations and attitudes can change. Make more good SEO visible, and maybe the spam won’t be the main thing that seems to speak for the industry.

I guess Derek Powazek isn’t done with attacking SEO. Now he’s published an SEO FAQ page which, sorry, really doesn’t provide much FAQ about SEO. So what the heck. I’ll deconstruct it. Be sure to also read my previous post, An Open Letter To Derek Powazek On The Value Of SEO.

What’s SEO?

First, any good FAQ ought to define what the subject is, right? So what’s SEO? Turns out, we have an SEO page here on Search Engine Land that explains this succinctly.

SEO — search engine optimization — is the process of getting traffic from the “free,” “organic,” “editorial” or “natural” listings on search engines. All major search engines have such listings, where web pages, web sites and other content such as videos or local listings are shown and ranked based on what the search engine considers most relevant to users. Payment isn’t involved, as it is with paid search ads.

You can read more on our page, along with some links to past coverage, a guide from Google and additional information

Notice in the definition above, I’ve mentioned things like videos or local listings. Search engines, and SEO today, aren’t just about web pages. That’s why when Derek rants that SEO is just some web developer’s job, he exposes a real lack of knowledge of the current state of SEO itself. In turn, frankly, he’s not giving out good advice.

SEOs Are All Scumbags

You know, I went to the gym today. Exercised with a personal trainer. All that trainer did was do stuff with me that should be common sense if you read a book, looked at the instructions on the machine and so on. Why isn’t he some scumbag that’s ripping me off? He’s a professional. He knows exercise intimately, in a way that I do not. It is his job to be an expert in this area. I want that expertise, someone who is totally dedicated to that area, not someone who dabbles in it.

SEO is a profession. Companies from MTV to the New York Times to the Wall St. Journal to Yahoo, to name only a few, employ full-time people who are responsible for SEO (and they aren’t scumbags, either). These companies have some of the best content in the world. And yet, they can still have major issues in how their sites are built or written or constructed that prevent them from doing well in Google or other search engines.

These SEOs, by the way, struggle with web developers who “think” they know SEO but don’t. Web developers who think that despite what an SEO tells them, a 302 redirect is the way to go. And thus the International Herald Tribune loses thousands of links because who wants to trust the scummy in-house SEO, right? I’ve got story after story of web developers and designers who think they know SEO but don’t, who cause major problems for web sites, and yet NO ONE ever writes a blog post blasting them.

There are bad SEOs. There are scummy people who there who spam blogs, who try to go black hat against search algorithms, who figure all’s fair in the fight for traffic. There are also bad people in any industry out there. You don’t count out an entire industry with helpful people because of your outdated ideas of how things work on the web. Especially if only deal with a particular type of web site (Derek doesn’t, for instance, seem to deal much with shopping sites, as best I can tell).

Deconstructing Derek’s FAQ

With that setup, let me go through the “FAQ” points Derek’s put out there and comment on them:

I publish a magazine and I know a lot of magazine publishers. And they are forking over embarrassing sums of money to charlatans who say they can raise their search engine rankings. These magazines can barely pay their writers. That’s wrong and it has to stop.

If you’re a company that’s about to pay some SEO expert, please, I beg you, take that money and hand it to a talented writer or competent web developer instead. It’ll be much better spent.

If you’re a company that hasn’t reviewed your site for SEO aspects, I beg you, consider this. Unless you like the idea of potentially wasting money, it pays to ensure you’ve done the basic things out there that will give you traffic for free. Maybe everything’s all perfect already, that your web developer who also knows some things about SEO has it all down. But it’s cheap insurance for many sites with a budget to ensure they’ve implemented SEO properly. Or, you can buy AdWords and pay by the click for that traffic from Google.

As for charlatans out there, yes, there are some. Just like there are some bad writers, bad designers and bad web developers. Ask for references. That’s not rocket science.

But I use SEO for good. Then you’re called a Web Developer. Good web development includes using proper formatting (like putting headlines in H tags) and understanding how the web works, search engines included. Valid code also has the side-effect of making your pages more accessible for your users, which is the point. Making your pages more accessible to robots is for robots.

Hey, I can fire up WordPress using the Thesis theme and make a nice site. Am I a web developer now? No need to pay some web developer hundreds of dollars per hour. I’ll do it for you as well.

Web development is not SEO. Good web developers will understand the fundamentals of SEO, in terms of good site architecture, crawlability and so on. But they probably won’t be ensuring that authors of a site are tapping into keyword research tools to ensure that when they write an article, they’re using terms that an audience seeking that article might use.

More important, few of them are dialed into how to handle giving Google and others a shopping feed. Or a feed of real estate listings. Or the completely separate ranking aspects that impact YouTube (the world’s second most popular search engine). Are they putting out a full-feed that Google Blog Search prefers? Are they checking that the URL shortener you use on Twitter spits back a 301 rather than a 302 redirect or worse, frames stuff up via a 200 code?

This isn’t 1999. This isn’t put stuff in a H1 tag and you’re good. This isn’t a world where we lack important tools such as the ability to feed sitemaps or even today, a way to see pages exactly as Google sees them. SEO encompasses any type of search engine dealing with any type of content.

[Insert irate defense of SEO here.] You sell SEO, right?

Yes, some of the people who are defending SEO also either perform it or are related to the industry. And some of those same people will readily admit that the industry has a terrible reputation. But then again, they actually KNOW SEO. They’re responding because unlike people like Derek or others who launch rant posts like his every six months or so, they deal with confused people all the time. They pick up the messes that web developer make. They solve the real problems out there that just being “real” won’t help. The don’t deserve to be written off. They actually deserved to be listened to, because they have lot of valid things to say.

If you’re so smart, prove it. I’m not that smart. But I can say that I am Google’s third result for “Derek” (I was number one until Wikipedia came around). And after less than 24 hours, my post about SEO is the ninth Google result for “SEO”.

Well, if Derek’s SEO strategy was to find people and convert them into sales for those seeking “Derek,” congrats. But actually, part of SEO is understanding what a target audience is after, in terms of your content. And I don’t care who are. Unless you’re doing a personal blog with absolutely no particular audience in mind, you do have a target audience. You’d better understand how that audience might be seeking your content. Otherwise, you actually might not get found. Good SEO starts with good keyword research or at least an awareness of it.

As for Derek ranking of SEO, welcome to the world of “query deserved freshness,” where Google will reward fresh posts with a relevancy bump. Does your web developer understand that? Because if they don’t, then when their boss starts complaining in a week how your great ranking just went away, they might start messing with your site to regain it — and make things worse.

How did I accomplish these magic SEO feats? Exactly how I said I did: Make something great. Tell people about it. Do it again. I’ve been doing that (or at least trying to) since 1995.

In this case, I wrote a passionate post. I posted it to Twitter and Facebook. And I sent one email to a friend about it. That’s it!

Sure, Derek is passionate about an industry that has a lot of haters. Guess what. I’ve been writing about SEO since 1996, telling people the good ways to do it and why it helps. And you know, it HAS helped lots of people.

If SEO is just all snakeoil, why am I still here? Why is SEO still here? Derek is hardly the first to have a rant. I’m easily into double-digits of rants like his I’ve read over the year.

The answer is that it ISN’T all snakeoil. SEO really is and can be useful. It’s just a pity that more people who do get helped by it who aren’t SEOs themselves don’t speak up for it. But here’s another tip. If your company does well with SEO, you usually don’t want to talk about it — because you’re hoping your competitor is listening to people like Derek who say SEO is all bullshit.

SEO is needed because of bad web design. Wouldn’t it be better to make competent websites in the first place?

Yes, it would. And when the miracle happens when all web developers are doing that, please let us know. In the meantime, perhaps instead of antagonism, we can talk about ways that a good web developer can identify a good SEO to work with? And also understand that even then, SEO extends beyond web pages these days.

It’s easy for you – you’re an expert. I’m using WordPress (which is free software) and the DePo Skinny theme (which I designed and released for free). You can download both of those and have a site just like this in a few minutes. No need to pay anyone for SEO. Then comes the hard part: Spend over a decade making things. If you do that, you’ll be an expert, too.

So just be an expert in selling ranch fencing, and you’re good. Or one on hazardous waste disposal in your local community, and you’ll rock. Or toss up a blog about your local locksmith site, and you’ll be cooking along. Or not (see Searching For Small Businesses, Coming Up Frustrated).

Passion is great. Wonderful content indeed rocks. And even “boring” businesses can tap into viral popularity. But you also have some boring SEO stuff you probably want to attend to, as well.

[Insert personal attack here.] I may have tarred a so-called “industry” but I didn’t attack anyone personally.

Really? Let’s review some statements:

Search Engine Optimization is not a legitimate form of marketing.

If someone charges you for SEO, you have been conned.

A new breed of con man was born, the Search Engine Optimizer.

SEO cockroaches employ botnets, third-world labor, and zombie computers to blanket the web with link spam.

SEO bastards are behind worms that attack blog services like Blogger, WordPress, and Movable Type.

It’s a game, and every link is a score for the SEO jerkwads and their disreputable clients.

Yeah, that’s a personal attack. Derek is calling each and every SEO out there someone who is not legitimate, someone who cons, someone who apparently runs botnets and worms and only has disreputable clients.

None of that’s qualified. He points it at each and every person who specializes in SEO. Part of me is amazed more people haven’t emerged to call him out on this type of hateful, unfair and inaccurate attack. But part of me figures these days, plenty of big voices hear SEO and either don’t want to speak up for them or based on their experiences as blogging rockstars think that everything that works for them must work that way for everyone else.

This may be obvious to you, but it’s not to everyone. You’re right. I regret saying that this stuff was obvious without explaining what I meant. Here’s what I meant: Good SEO techniques are just good web development techniques. They should be obvious to anyone who makes websites for a living. If they’re not obvious to you, and you make websites, you need to get informed. If you’re a client, make sure you hire an informed web developer.

So I heard Chris Brogan speak recently, and he joked that his book Trust Agents (cowritten with Julien Smith) was just full of common sense. Which is true. But what’s common sense to someone who knows their field is uncommon wisdom to those who don’t. Some SEO techniques are also good web development ones. A good web developer should know them or should educate themselves. But plenty do not. Or plenty assume they know it all when they do not.

CLIENTS: If someone approaches you about optimizing your search engine placement, they’re running a scam. Ignore them. If your site isn’t showing up in Google, fire whoever is making your web pages and hire someone better. Sign up for social media services (Twitter, Facebook, etc) and participate there. Pay for quality writers and designers – that’s what will actually raise your ranking in the long term.

Agreed, if someone approaches you out of the blue — you get that email offering to do some analysis, etc. Ignore it. Might not be a scam, but it is generally not a sign of a company that’s in demand because of a satisfied client basis. As for firing your web page builder, um — maybe. Or if you like what they’ve done with the site, and SEO might be able to work with them and allow you the best of both worlds.

WEB DESIGNERS: Learn to code your own pages. If you can’t, hire someone who can, and listen to them when they tell you why putting all that text in an image is a bad idea.

Those people who can, by the way, are called SEOs. You know, those scumbags you shouldn’t be listening to.

WEB DEVELOPERS: Educate your designers about proper web development. Educate your clients about how the web works. Follow Google’s advice. Read A List Apart. Writing good code won’t just help your Google rank, it’ll make certain your site is accessible to screen readers, mobile devices, and all the browsers out there.

Well, I don’t regularly read A List Apart, so I can’t comment on the quality of the SEO advice over there. But yeah, read Google’s. In particular, read their SEO starter guide here.

You might take in a conference or two. At our SMX East conference last week, we had an entire track devoted to technical SEO and developer issues. Want more? Jane & Robot started last year is devoted to developer & SEO issues, and they have events.

You can check out topics here on Search Engine Land, such as our 100% Organic column and our Google SEO, Microsoft’s Bing SEO, Yahoo SEO and How To SEO sections.

SEO SPECIALISTS: If all you do is SEO, you need to expand. Hire a visual designer and some kickass coders and become a real web agency. Start making sites good from the get-go instead of cleaning up other people’s messes. Besides, if all you do is SEO, your days are numbered. Social media is rapidly becoming much more important than Google. (Number one referrer to my site this week? Twitter.)

Again, this isn’t 1999. There are plenty of agencies that offer SEO services. Some of these were pure SEO companies that grew. It’s also no newsflash to many good SEOs that social media has grown in importance. In fact, the real news seems to be that Derek doesn’t realize they already know this.

GOOGLE: Would you look at all the crazy you’ve created? You could fix this by making your engine more inclusive of websites with common mistakes, introducing some randomness to the order of results (why should everyone’s results be in the same order?), and unforgiving punishing the businesses that have sprung up to exploit your popularity. Get on it.

Yes, they do. That’s one reason why Derek’s ranking for SEO at the moment. It’s also the reason that plenty of people like him can say you don’t need to do SEO. Because for lots of them, especially if they have promient blogs, Google does do the right thing.

But it’s not perfect, and you can help it. That’s why over the years, it has rolled out a ton of tools at Google Webmaster Central that any good SEO will know about.

Complaining that Google should just “fix” everything also reminds me of web developers who have long assumed that problems with search engines and Flash would just eventually go away, so they didn’t need to do anything. I’ve heard that for over 10 years. Despite advances, the basic advice remains the same. Try not to lock your content up in Flash.

When developers will test their pages to make sure they work in Internet Explorer, Firefox, Safari and Chrome, not testing against the biggest browser of them all — Google — is foolishness. What Derek’s written above is the same as complaining that Internet Explorer has some bug that’s screwing up his web pages, and rather than fix it, he thinks IE should solve the problem. Sure, IE should — but in the meantime, you workaround it.

Postscript: Let me make clear my intent in this piece isn’t to attack Derek personally. I have issues with many of his statements. Since these statements seemed based on his personal experience, it’s difficult to avoid addressing him on this by name. I’ve read through this piece again, and I really don’t see a personal attack him Derek. On Twitter, in reference to his original post, I did say things like:

yes, let’s make heroes out of people who rant ignorance & harm others who do good work. hurray! (here)

& sadly plenty of SEOs provide good non-sleezy, useful, legit, google-backed services but get tarred with usual ignorant brush (here)

see my tweets earlier today. it’s just the latest in years of ignorant slamming of SEO but (here)

I didn’t mean a personal attack by calling what Derek wrote a rant. That’s just a fact. It was. He called it the same himself. Heck, I rant all the time. There’s nothing wrong with a good rant, though I prefer one that’s also thoughtful and backed up by facts.

That’s where the ignorant part is. I don’t think Derek is ignorant or stupid in general. But I do think much of what he wrote seems to be ignorant of the current state of SEO. Ignorant as in not aware. A better choice of word, which might not have seemed perhaps personal, would have been uninformed.

Anyway, hopefully Derek and I will talk more about this tomorrow rather than playing dueling posts. My intent was never to attack him as an individual, but I certainly wanted to raise some awareness and balance to the issues he put out there.

Derek Powazek launched an attack on SEO yesterday that really said nothing that others haven’t ranted about before. I’ve responded to many of these attacks over the years in hopes of educating people about mistaken assumptions. I’ve largely given up. But I figured this time I’d give it another go with some personal illustrations I’ve encountered recently.

In particular, rather than do an article to counterbalance Derek’s post, I started to write him a private email. But as I composed that, I felt it might better illustrate to everyone why SEO is indeed a legitimate form of marketing and those who provide the service are not all “scammers” who are out to “con” you.

So Derek, I saw your rant, and it was disappointing. Your post was based on “14 years of hits and misses.” Well, my response come from my own 14 years of covering search engines. Of having answered feedback from hundreds of people. Of having talked with hundreds of people personally. Of understanding that the “you just build it; you just put it out there” approach to search engines, sadly, doesn’t always cut it.

Let me be clear. I totally agree with your core advice. Build a site for visitors. Have great content. These are the keys to success, not just with SEO but with anything you want to do. In fact, we just had an article on our site here reinforcing this.

Still, sometimes people have problems. And the stuff that you think isn’t rocket science — that anyone knows — is indeed a mystery to others. They want help, and sometimes they can’t find that web developer who also understands SEO issues. In the same way, you sometimes don’t find web developers who are also designers. Or designers who understand conversion issues. Or conversion experts who understand web development.

Let me tell you some stories.

A Mother Who Sells Homes

Two weeks ago, our local elementary school had a mixer for parents. I was talking with one woman who asked what I did for work. “I write about search engines,” I told her. That led her to asking if I know about how people get found on Google. Yep. So she started asking about her local real estate site, and how she might market it, things that she might do.

That’s a loaded question. See, for you, she just needs to believe in the real estate she’s selling, then tell her friends with personal notes, get out on Twitter, find places where her community congregates and “be real.”

But to really be real, let’s remember that she’s selling real estate in one of the most competitive areas of the country, Newport Beach, California. Her friends aren’t all going to buy homes she’s listing. Her “community” congregates on Google and does things like type in “newport beach homes for sale.”

To succeed in attracting that audience, she should have a great site and great content — agreed. But does she have individual listings? Then she probably needs to kick them out into Google Base, in order to fully be listed in Google. Does your mythical web developer deal with Google Base much? And where’s her web site now? Is she running it off Blogger? Using her own domain? These have impacts on how both the search engines may see her as well as how she’s perceived.

Does she have a blog in addition to a main site? That has an impact. Has she considered some unusual, creative ways to create content around real estate in her area, perhaps some catchy link bait, which may pull in the links she needs to rank better (which, by the way, is a recommended Google practice).

Does she have a local office? If so, has she claimed her listing in Google Local? If so, has she updated her title to reflect that perhaps she has “newport beach homes for sale?”

This is all SEO. It’s not your father’s (or mother’s) SEO that you rant about, the keyword stuffing, the link drops (none of which is best practices SEO anyway). But make no mistake, it’s SEO.

SEO & Baseball Practice

Later that same week, I took my son to baseball practice. I also took my computer, so that I could finish up work on an article and have the weekend clear when he was done.

Another father came over to me, asking what I do. “I write about search engines….” Which as before, led to the questions about how his company might show up on Google. Did I know much about that?

Sure. And since I had my mobile broadband card with me, I fired up his web site. Very sad. Same page title on every page. No keyword research employed, to think how people might be seeking out the industrial shipping cases they sell. Long, dynamic URLs that might pose indexing issues.

Where do you start with someone like this? In your world, his company should just “be real.” He sells industrial shipping cases. How real do you want to be about that?

His “community” are the people who realize they need a case like he sells and go online to places like Google and start searching for the products. And his pages are NEVER going to show up, because there’s nothing unique about his site and he had basic SEO errors that haven’t been dealt with.

If all he did was change things so that his page titles were different, he’d pull traffic. I know, I know — that’s so obvious. But it is NOT to him. His job isn’t to do web development. He’s not you or I with 14 years of having learned all this stuff along the way. He actually deals with things like ordering the products, overseeing workers and doing an array of offline marketing.

Meanwhile, his web developer clearly did NOT have any SEO thoughts in mind when building the site. That’s all too common. If I wanted to be snarky, I could do an entire post on why web developers are a waste of time and you should just employ SEOs who can also build web sites. But the reality is that a good SEO (and there are some) working to help direct the web developer could solve the site’s problems quickly. It’s called teamwork, and it’s awesome when it happens.

Mommy Bloggers & SEO

Over the weekend, I had a group of mommy bloggers over at my house. Not because I’m a mommy blogger but because my wife is a mommy blogger. The group was here because they’re all involved in a new project for mommy bloggers in Orange County.

My participation was to show up at the end of the meeting, not to say anything but just because I was coming home. I caught some of the closing conversation, and a question about Google came up. How was the search engine going to deal with their content, if these mommy bloggers put the same article on a publishers site as well as their own sites.

Answer? I don’t know. Google might decide to favor the publisher’s site not because it’s “more real” than the mommy bloggers but because it has built up more authority collectively over time. Certainly the mommy bloggers themselves are all “real” in what they write, how they put themselves out there to their communities and so on. Much depends on what they’re most concerned about.

If they had a key post that they absolutely wanted to be the primary source for, guess what? They’d better have some SEO savvy. They might need to tell the publisher to prevent the copy of their posts they provide from being spidered on the publisher’s site. Or if they allow it, then they might want to make use of the forthcoming canonical tag 2.0.

That’s not being “real.” That’s SEO. And that’s SEO that a good SEO will know — and many non-SEOs will not.

I’m glad, honestly glad, that you’re savvy enough to understand how good SEO can be incorporated into web development. I wish more web developers could do the same. But my experience has been that much good SEO gets overlooked. There are bad SEOs out there, who give the entire industry a bad name — just as there are bad bloggers, bad designers, bad cops, you name it. There are also excellent SEOs who work inside of companies as well as through agencies for hire. Don’t tarnish an entire industry that actually helps many, many people in ways I’m sure you would agree with.

For those who just can’t get enough of this subject, some of my past writings on the topic. They cover plenty of additional examples, plus the fact that Google itself recommends SEO:

• Dear Fox News: SEO Is Not Search Engine Scamming (Unless You’re Scamming Yourself): Aug. 2009
• Importance Of SEO Makes Front Page Of Los Angeles Times: March 2009
• A Bad Month For SEO’s Reputation: Feb. 2008
• From My Inbox: More Defense Of SEO: May 2007
• SEO: Real Skills That Can Protect Your Traffic: March 2007
• Why The SEO Folks Were Mad At You, Jason: Feb. 2007
• More Rounds In The “Is SEO Overrated” Debate: Jan. 2007
• Defending SEO, Yet Again!: Dec. 2006
• Yes Virginia, SEO Is Rocket Science – Defending Search Engine Optimization Once Again: Dec. 2006

See also the follow-up to this post, SEO FAQ That’s Not From The Land Of Unicorns.

And then there were none. Yahoo has long been the only major search engine that supported the meta keywords tag. However, the search engine revealed today that like the other majors, it no longer supports it.

The news came during the Ask The Search Engines session at SMX East in New York today. The search engines were all asked about their support of the tag. Moderator Danny Sullivan noted that only Yahoo provided support of the tag — prompting Cris Pierry, senior director of search at Yahoo, to announce that support actually had been ended unannounced “several” months ago.

Bing doesn’t support the tag. Google has never supported it and in fact clarified this again in a special post last month. See Google: Stop Suing Over The Meta Keywords Tag, We Don’t Use It for more about that.

NOTE: See our follow-up post, Sorry, Yahoo, You DO Index The Meta Keywords Tag.

Search and online marking firm 360i published findings from a small but structured test of how the “Caffeine” search infrastructure changes could affect results and ranking on Google. As Vanessa Fox wrote when Caffeine went public earlier this month, there apparently would be a number of changes in the ordering and content of results. As she observed in that post, “Google Caffeine will cause quite a kerfluffle in the web developer and search engine optimization world and many will dive in to try and figure out the changes.”

This is precisely what 360i has now tried to do. The company evaluated 40 retail-oriented keywords/searches to compare results before and after, and the potential SEO implications of Caffeine:

[A] sample set of 40 retail keywords. We looked at ten major retail brand names (keywords), ten retail head terms (single keywords), ten retail torso terms (two-word phrases) and ten retail long-tail phrases (four-word phrases) and compared the search results on the first three pages of both engines (standard Google and “Caffeinated” Google).

With the caveat that “everything could change” before Caffeine formally rolls out 360i offered six observations or “things to expect” if/when it does. Here they are:

1. Domains and rankings will fluctuate.
2. The index size, or “competition,” of single keyword search relevance will increase.
3. You’ll see a boost in relevance for long-tail searches.
4. You’ll get results (SERPs) in half the time, on average.
5. Blended results will increase.
6. There will be a social jolt.

To read the larger discussion of each of these takeaways go to the 360i blog post.

72 search engine optimizers from around the world have contributed to SEOmoz’s latest Search Engine Ranking Factors report, a study that offers valuable insight into how SEO works in 2009 … but also shows how little it’s changed in recent years. (Disclaimer: I’m one of the 72 contributors.) You can read the survey here, and there are also some interesting comments on SEOmoz’s blog post.

Participating SEOs completed a lengthy survey earlier this year that involved ranking possible factors that Google uses when ranking sites. The survey also included a variety of questions about current SEO trends and topics.

According to this year’s survey, the top five ranking factors are:

1. Anchor Text from External Links
2. Keyword Use in Title Tag
3. Raw Link Popularity
4. Diversity of Linking Domains
5. Keyword Use in Root Domain

Those first three factors are unchanged from the 2007 version of this survey, although the order is slightly different. Two years ago, the top five were:

1. Keyword Use in Title Tag
2. Global Link Popularity of Site (”Raw Link Popularity” above)
3. Anchor Text of Inbound Links (”Anchor Text from External Links” above)
4. Link Popularity within the Site’s Internal Link Structure
5. Age of Site

It looks like SEOmoz changed the structure of this year’s survey a bit. Negative ranking factors are presented separately, and there’s a new chart of “Overall Ranking Algorithm” elements. In fact, the fifth factor on the 2007 list, “Age of Site,” is now essentially the same as (or at least very similar to) the number one piece of the Overall Ranking Algorithm, “Trust/Authority of the Host Domain.”

When it comes to the big picture of SEO, the primary factors really haven’t changed much over the years. But what separates “good” SEO from “great” SEO is often in the details. That’s where you’ll find the true value of the SEO factors survey — in learning the little things that competitors may have overlooked.

In the same month that Rupert Murdoch has said he’ll charge for all his news sites because “quality journalism is not cheap,” I find it ironic to read the rubbish about search engine optimization that Murdoch-owned Fox News “reported” this week.

SEO equals scamming, we’re told. If I have to pay for this type of article from Fox News in the future, I sure hope there’s a refund policy. Below, an open letter to Fox News encouraging them to correct their error, lest they want to be considered scammers themselves, since they practice SEO.

The article from August 17 is called “Top Online Marketing Jobs To Leave You Friendless” and consists of a series of “slides” that cover different jobs. One covers SEO, where we’re told:

Search Engine Optimizer

Ever wonder why “nonsense” Web sites sometimes turn up in your search results on Google or Yahoo? That’s because search engine optimizing scammers work full-time to create thousands of other Web sites that link to the spam site. For example, the creator of spamlaw.com is hoping to dupe would-be visitors to spamlaws.com, a legitimate site that bills itself as an online security resource.

Source: FOXNews.com

See, “search engine optimizer” is considered the same as “scamming” search engines. Fox, of course, actually means spamming search engines. But that’s one of the many things they get wrong

First, the example given isn’t SEO at all. It’s called typo-domaining. Someone owns a domain name that’s a typo or similar to a more commonly known domain name. They put up a page stuffed with paid links, in hopes that people who accidently type in the domain name into their browsers reach the site. Search engines aren’t involved — this is trying to catch people directly from their browsers. It is also NOT the same as domaining in general, where people may try to earn money off catchy domain names that are not typos of other brands.

Next, on to SEO itself. Search engine optimization is NOT spamming search engines with junk. It is the process of ensuring that a web site can be found on search engines. It’s legitimate job, which many people undertake within all types of companies. It is something that Google endorses and recommends, either that people do it themselves or work with good companies.

If Fox News had done any type of reporting, they’d have quickly learned this. I know they’ve talked to at least one search expert I’m familiar with in the past. But given the source for these “facts” is simply “Fox News,” no reporting at all was apparently done. Someone at Fox seems to have written up whatever they believed or heard, no reporting required.

Fox News apparently didn’t even bother to talk to their own person in charge of SEO. Perhaps because the story is true, and the Fox News SEO like all SEOs, according to Fox, is friendless.

I know someone at Fox News is doing SEO because if I look at articles, I see the use of the meta description tag. In fact, on the page slamming SEO as scamming, I see this:

<meta name=”description” content=”Job-hunting? Think about becoming an e-mail spammer … or a Web site spammer … or a search engine optimizer. Here’s a great opportunity to become part of a team of Web-savvy professionals who clog the Internet with unwanted ads and sell users’ personal information to the highest bidder. Not only are these jobs legal, they can be downright lucrative. Here are some of the top online marketing jobs that will make you money . . . and leave you alone and friendless.”/>

The meta description tag is a long-standing SEO tool designed to help site owners control how they are described in search engines. It is classic SEO. Google offers advice about it. For it to be on Fox News pages means they practice SEO. So according to their own article, they’re either scammers or they’re reporting things inaccurately.

Over at the robots.txt file for Fox News, I find this:

User-agent: *
Disallow: /printer_friendly_story
Disallow: /projects/livestream

#
User-agent: gsa-crawler
Allow: /printer_friendly_story
Allow: /google_search_index.xml
Allow: /google_news_index.xml
Allow: /*.xml.gz

#
Sitemap: http://www.foxnews.com/google_search_index.xml
Sitemap: http://www.foxnews.com/google_news_index.xml

The robots.txt file is a long-standing SEO tool for deciding what search engines can index from a web site. For Fox to have one means someone is practicing SEO. Moreover, the last two lines indicate that Fox News is explicitly generating sitemap files to feed to Google, something that is done to increase the chances of showing up in Google’s search results.

It’s SEO — being done by Fox News, and so again either Fox is scamming search engines (with nonsense like this article on SEO) or has made an error in reporting that needs to be corrected.

I’d mentioned that Fox really meant spamming, not scamming. Spamming is when someone goes beyond the accepted guidelines of a search engine to try and gain a ranking. A very, very long time ago, terms like “spamdexing” where used to describe this. But in another slide, we get treated to this:

Spamdexer

That Google search may seem reliable, but spam can be hidden within those top ten results. A common technique by a “spamdexer” is to include keywords like “health care” at the bottom of their Web page to boost search results. But instead of getting the legitimate site you hoped for, unsuspecting users will see sites masquerading as the real thing.

Source: AP

I don’t know anyone who runs around calling themselves a spamdexer. And I’d look up the reference more, but despite the source being the AP, Fox doesn’t link to the original content.

Spamming has many forms and activities, and hidden text is indeed one of them that should be avoided. It is also largely an outdated tactic. Search engines rely far more heavily on analyzing links to determine rankings, which brings up another irony.

Scroll to the bottom of the Fox News home page. See those ads for CarsDirect, BankRate, and People magazine among others? They each have links in them. For example, the People link says “Celebrity News.”

Since these are ads, those are paid links. Google has a strict policy that sites that sell paid links should block them from passing credit (Fox doesn’t — you can see how they show up at the bottom of this cached page). Otherwise, a site like Fox News is effectively selling some of its reputation to help other sites like People rank well for the words in the links (and surprise, People ranks number two on Google for “celebrity news”).

A good SEO would be able to advise Fox to start blocking links like that, lest Fox be deemed to be scamming and spamming Google, which in turn might cause the home page to drop from a highly regarded PageRank score of 8 to something lower. That change potentially could cause Fox News content not to do as well in Google, as a penalty for spamming.

Fox, block your links. And more important, correct the article and issue an apology.

Postscript: Via SEO Book’s write-up of the Fox News article, I came across this long interview about Fox’s SEO efforts — multiple people, each assigned to a different division. Wonder how they feel about Fox News calling them friendless scammers? Though, this is about Fox Interactive’s efforts, and they don’t oversee Fox News, Natasha Robinson points out. Still, it also makes me think that the Fox News editor and writer involved with this story lacked SEO training — bad news for a site that hopes to bring in some traffic. Perhaps they should see my Quick Tips For Newspapers & SEO post.

Microsoft recently published a PDF about Search Engine Optimization called Bing: New Features Relevant to Webmasters. This is the second SEO-related offering in as many weeks. During SMX Advanced, Microsoft launched the IIS Search Engine Optimization Toolkit. I’m working on a review of that as well, but it requires Vista or Windows 7 and IIS 7, so installation takes a bit of effort. I talked with Microsoft about the information in the PDF and got some clarity and additional information. The PDF primarily describes the user interface changes launched with Bing, and how those might impact site owners, but also touches on search engine optimization.

Google has provided SEO advice for a while, both in their help center and in their Search Engine Optimization Starter Guide, so check those out if you’re interested in hearing the official search engine stance on SEO. Google’s document is much more instructive, providing their top guidelines for creating websites that can be easily crawled and indexed by search engines.

What can we learn from Microsoft’s advice for webmasters?

The document starts by describing the primary difference in Bing vs. Live Search that impacts site owners: the first search results page no longer shows results 6 through 10 for the searcher’s query. Instead, the page includes “categorized” results after the list of 5 sites that match the initial query. So, for instance, if you search for [vw beetle], results 6 through 8 are for [volkswagon bettle for sale] and results 9 through 12 are for [used volkswagon beetle].

Expanded long tail opportunities

The document clarifies that more than 300 factors are used in determining this categorization and that what categories appear is entity-dependent. In other words, if you search for [buffy the vampire slayer], you may see a category for “wallpaper” because lots of people search for Buffy wallpaper, but you might see categories for “episodes” and “characters” because Microsoft has identified “Buffy the Vampire Slayer” as a TV show.

Microsoft says that this new way of organizing results provides new long tail opportunities for site owners. They suggest determining what entities you most want your site to rank for and then optimizing for the long tail queries within those entities. For instance, if your site is about cars, you’ll want to optimize for the car brands (VW, Volvo, etc.) but also car-related categories such as dealers, used, and for sale. That will give you additional opportunities to rank on the first page of results.

Best match

Bing has also introduced a concept called “best match”. When Bing is reasonably confident that the first result is what the searcher is looking for, it will be noted as the “best result” and could contain additional details, such as a box for searching within the site, phone numbers, and links to pages within the site. In some instance, no results other than the best match appear on the page, which Microsoft says their team chooses when they have high confidence in the result and the query volume is high. Of course, this isn’t great news for the site owner at position 2.

If you don’t want your site to appear as the best match, don’t want the internal site search surfaced, want to correct the phone number that appears, or want to remove links, you can request that by emailing bestmatc@microsoft.com.

Document preview

Document preview provides additional content from the site in a hover and, in Microsoft’s words, “helps searchers find the content they want faster, without leaving the SERP until they are ready”.

They say this helps increase qualified traffic, but some webmasters might think that it helps searchers find the content they want without leaving the SERP at all. For them, Microsoft provides a way to opt-out of the feature. Just add the following to the <head> section of each page:

<meta name=“msnbot”, content=“nopreview”>

If you don’t want this feature used on any page of the site, it might be easier to return the directive in the server HTTP header as follows:

x-robots-tag: nopreview

Not everything from the document preview comes from the site. Some information could be from external sources as well. For instance, if the site is in Flash and Microsoft has trouble extracting data, they might turn to a third-party source. And they may use local information such as an address or phone number from an external source. You can request that third-party data not be used for your listing by emailing bestmatc@microsoft.com.

Flash extraction

Microsoft says that Flash-based sites are responsible for 21% of all empty descriptions in their index. They say they are doing “limited data extraction” and are now able to generate descriptions for one-third of those. They may also use anchor text from incoming links. Of course, Google has been using incoming anchor text as source data for missing titles for a while, and has continued to evolve its ability to extract Flash data. No word on whether Microsoft is using Adobe’s crawler API or another technology for this extraction.

Use of microformats

While the document encourages the use of structured data, such as microformats, Microsoft tells me that they aren’t currently using this data for crawling, indexing, or ranking. This stance mirrors Google’s and Yahoo’s, both of whom are encouraging structured data use, but are not yet using the data for web search.

Instant answers

Instant answers, the “OneBox”-style results that provide data to answer the query, aren’t new to Bing, but may be gaining prevalence. For now, Microsoft tells me that you can request evaluation of your data for inclusion using their support forum.

Local listings

Local results are more heavily featured in Bing. You can add your site to their Local Listing Center to ensure you have the opportunity to be featured in these results.

Searcher behavior

Microsoft provides some interesting data about searcher behavior, which they tell me is based on a number of internal studies derived from analysis of their search logs and toolbar logs. For instance, they found that searchers refine their query, bounce back to the search result, or abandon the search 50% of the time. The breakdown of that 50% is below.

They also found that searchers repeated 24% of their queries during a session (which led to the additional of search history in the left column of the search results page).

All in all, this document doesn’t provide a lot of new information about SEO. But I applaud Microsoft for understanding things like removing results 6 through 10 and replacing them with categorized results and adding a hover with additional content from the site could impact search traffic and for providing information about these features. It’s difficult to say just how much these changes will effect traffic patterns. The biggest change, of course, would come if Bing brings about Microsoft’s goal of increasing search share.

What follows is a rant, which is something I rarely, if ever, do. It’s done in the spirit of fun, so don’t take it too seriously. Enjoy!

I feel like the grandpa who laments in a crotchety voice to his grandkids: “Nobody ever writes letters anymore! They just sit on their computers and their cell phones all damn day!” But instead I’m saying: “Nobody ever blogs anymore! They just tweet and re-tweet!”. For example, this tweet by @dannysullivan could have been a fantastic blog post. Instead: it’s 129 characters that merely hints at the story:

@dannysullivan: seriously, pinkberry with locations in 2 of 50 states ranks 14 for yogurt? http://bit.ly/dHOYe well @mattcutts does love them :)

Last week I had the pleasure of sharing the stage with Danny on a panel at the eMetrics Summit. The topic, unsurprisingly, was SEO, but targeted to web analytics geeks (a number of whom were SEO newbies). Danny kicked off the session with a quick SEO 101 where he expanded on his gem of a tweet above about Pinkberry.com. Pinkberry is a frozen yogurt brand that I was unaware of until the session. And what a brilliant example it was. Pinkberry.com is a case study in how NOT to build a website. I think they hired the Anti-SEO to ensure they wouldn’t rank for anything other than their brand name.

There was really silly stuff going on. Basic, basic on-page SEO was completely mucked up. Like for example, the page titles. Danny showed the audience site: results in Google for Pinkberry.com and the results were, well, disturbing to say the least—at least for anyone with an SEO bone in his/her body! Sure enough, every title tag was the same across the site. But wait, it gets better! The titles were all one word long: “Pinkberry®”. Luckily, the major engines don’t trip up on circle R and TM symbols, even when they are ASCII characters, or I’d be complaining about that too! (Nonetheless, I dislike such symbols in title tags. If you must use them in titles or elsewhere or you get yelled at by your legal department, then please “escape” them, e.g. &reg;—it’s just good HTML etiquette.)

Let’s move on to what is on the home page, that most important of pages from an SEO perspective. It’s a circa late 90’s “splash page”. With, you guessed it, zero textual content. This is what the home page looks like from a spider’s perspective. Pretty sad. Well, to be more technically correct, this is what it sees: there’s a single image with no alt attribute and a filename that is of no help whatsoever.

Moving on past the content-less splash page, you end up on a page where the mouseover navigation relies on JavaScript, which of course the spiders don’t support. Not only were the mouseover nav items inaccessible, but the main buttons (the ones available without hovering) stopped working. At least the ones that had mouseover effects attached to them. This included their “Products”, “About”, “Contact”, and “Groupie Corner”. Oh, and again, no textual content to be found. But hey, at least they had defined some meta keywords, so clearly someone at Pinkberry is at the wheel driving their SEO “strategery” (*grin*).

I think the only thing the Anti-SEO didn’t do was take any textual navigation or content elements that may have been remaining in spider-accessible formats/locations and wrapped a Flash movie around all of them. And perhaps added frames for good measure, complete with hidden links in the frameset pointing back to His site.

Yet somehow, despite themselves (as Danny notes in the tweet above) Pinkberry ranks on page 2 in Google for “yogurt!” Huh? Or as the younger generation like to say: “WTF??”

Matt Cutts, care to comment? Is this a result of your hand editing since you’re such a raving fan? Or, put another way (by the more politically incorrect SEOs out there like DaveN), a “hand job”?

There is no doubt that you can learn a tremendous amount of information on search engine optimization (SEO) by reading sites like this or ones in our blogroll, but there is always a lot to be learned from getting your hands dirty. Now, you can get your hands dirty by experimenting and trying SEO techniques out on sites and you can also learn an incredible amount by trying to reverse engineer a web crawler by building your own.

In fact, Google Webmaster Analyst, JohnMu, tweeted this morning stating that fact. He said, “Want to learn about indexing/crawling? Don’t read – code a spider.”

That is exactly what SEOmoz did, they built a crawler and index of web pages to better learn about the internet, plus share that data with the industry. Linkscape was introduced in October 2008 and has grown to 44 billion web pages and 474 billion links.

Rand Fishkin of SEOmoz has posted the lessons learned from building an index of the web. So, maybe, in this case, reading about someone else’s experiences and findings in building such a crawler can help you.