For years, Google’s discovery of web pages was solely based on links. If a page had no links to it, Googlebot had no way of knowing about it and therefore, would never index it. Along the way, Google provided an option for submitting individual pages, but that wasn’t really a viable option for site owners with large sites. In 2005, Google launched XML Sitemaps, which was a much more scalable way for site owners to let Google know about pages of their site that Googlebot may not otherwise discover through links. Today, a Google Webmaster Central blog post discusses another way Googlebot may discover pages: feeds. They say that using RSS and Atom feeds to discover pages helps them learn about new content quickly.

New content is key for Google since freshness is a vital component of relevance for some queries. Convention wisdom is that it’s not all that useful to ensure Google knows about pages of your site if they don’t have links to them, because without links, Google won’t see them as valuable. But current ranking is much more complicated than the original PageRank formula describes. And new content with no links may very well trump content with an abundance of links if it makes sense for the query.

Of course, site owners have always been able to to submit RSS and Atom feeds as Sitemaps, but this post describes using these feeds even if the site owner hasn’t submitted them via the Sitemap system. Instead, they are scanning other feed submission systems, such as Google Reader and ping services for the feeds.

It’s unclear from the post if the feeds are being used solely for discovery or if the content from the feeds are being used in place of crawling as well. The title of the post references “discovery” but the post itself notes that they are able to “get these new pages into our index more quickly than traditional crawling methods” and to directly crawl feeds. If Google is using the feeds in place of crawling, this would be another argument in favor of full rather than partial feeds — you’d get more of a page’s content indexed more quickly. Google Blogsearch initially crawled feed content rather than the actual pages, which led to partial indexing in Blogsearch, but this changed late last year.

The post notes that in order for Google to use a feed as a discovery method, the feed must not be blocked by robots.txt.

Google Webmaster Tools has just launched a “labs” section, where you’ll find new features that may be early in the development cycle and not quite as robust as the rest of the tools. The features available so far are Fetch as Googlebot, which lets you see exactly what Googlebot is served when it requests a URL from your server and Malware Details, which shows you malicious code snippets from your site if it’s been flagged as containing malware.

Fetch as Googlebot

Of most interest to webmasters, SEOs, and web developers is likely the Fetch as Googlebot feature. You can specify any URL on your site and see the HTTP response (header and contents) that the server returns. Simply  indicate the URL and click the Fetch button. It may take a few moments for Googlebot to access the page and return the results, since it fetches the page in real time. (Refresh the page to see the progress.)

Click the Success link once it’s been processed to see the results.

How is this different from simply looking at the source code of the page?

• You see the HTTP header information at the top. This information is generally easily available through tools such as Live HTTP Headers, but isn’t contained in the source code itself (since that information is coming from the server, not the page).
• You can see if the server is returning any of the page information differently than the page has been coded.
• You can see if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (typically known as “cloaking“), you’ll see  what’s being conditionally served to Google.
• You can use the tool to test changes (particularly things like redirects) in real-time.

Note that this tool won’t necessarily show you the content that Google is able to extract from the page. If the page contains JavaScript, for instance, you’ll see the raw JavaScript code contained on the page, not the rendered view visible in the browser. Which, unfortunately means you can’t use this tool to determine if Google is able to access content contained in rich markup.

What’s this about cloaking?

This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re coming into a project late and aren’t sure what’s been previously done. It can also help uncover if your site has been hacked. Back in 2006, Googler Matt Cutts and I did a show on Webmaster Radio during which we talked about how in some cases, a hacker might add links to a site and then cloak those pages so that the site owner never sees them. Only Google does. At the time, Matt suggested using Google Translate (and choosing English to English) to see what Googlebot was being served, but this tool can now more easily serve that purpose. Matt confirmed this to me this morning: “The biggest use case is just debugging site issues. Of those, the biggest case will be hacked sites. Some attacks will hide content until search engines fetch the page (and some attackers add a noarchive tag so that the search result doesn’t have a “Cached” link), so a site could look clean to the website owner. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”

How do I test redirects?

If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. For instance, when I fetch www.searchengineland.com, I see that the redirect is correctly implemented as a 301 and points to searchengineland.com:

You can also use the tool to troubleshoot URLs listed in the Crawl Errors > Not Followed report. You can also test these URLs using something like Live HTTP Headers or by trying to access the URLs in a browser, but if neither of those methods uncover the problem, this tool can help determine that the issue is specific to Googlebot. You can also use this tool to verify that fixes you’ve made to redirect errors uncovered by the Not Followed report have really solved the problem.

(Note that the tool currently has a limit of 100kb per page. However, this is for the tool only and doesn’t apply to Googlebot’s normal crawl of the site. Google is monitoring feedback to see if many site owners find this size to be limiting.)

Malware details

The Google Online Security Blog has more information on the malware details tool. Previously, webmaster tools reported when the site was flagged has having malware and listed sample URLs. This new tool will also show samples of the malicious content, and in some cases, the underlying cause. This should help those site owners whose sites have been hacked to include malware find the problem and fix it. If your site does contain malware and you’ve fixed it, you can request a review to have the malware alert removed in search results.

Search engine optimization involves a lot of moving parts. Audience analysis, content optimization, rankings factors — it can be hard to pinpoint concrete, actionable data. But technical SEO is a little different. It’s certainly not any easier than the other components, but at least the data can be a bit more straightforward. You can check your server logs to see if the search engine bots haven’t been crawling of your pages (or have been crawling lots of duplicate ones with various URLs); you can generally determine if search engines are having trouble extracting the content on your pages (ever see “loading, loading, loading” as a snippet?); and while the search engines don’t come right out and tell you when they’ve tweaked a ranking factor, they do generally announce technical changes (”we can crawl Flash“; “we are executing onClick JavaScript events“).

And technical SEO is clearly of vital importance. If search engines can’t crawl your pages and extract your content, your site will never be well-ranked for the queries you care about, even if you have the most awesome content on the web and more backlinks than the Adobe Reader download page.

Historically, search engines have had a tough time with newer web technologies. They were built as text-based engines that searched HTML pages, so it’s been taking them some time to catch up with things like Flash and AJAX and video. But the times, they are a changin’. The last couple of years have seen some great improvements, particularly from Google. The new parameter handling feature of Google webmaster tools to help with canonicalization issues due to URL parameters is only the latest example in a long line of advances in this area.

Get the latest solid information

Search Marketing Expo East, coming up in just a few weeks in New York, provides (among many other topics) in-depth, actionable information about how to construct search friendly web architecture, diagnose technical issues, and be prepared for the next evolution in search engine handling of web technologies. I’ve programmed a number of sessions that I’ll be moderating and speaking at, along with some fantastic technical experts, and have put together a suggested itinerary for the whole show for those interested in the more technical side of SEO.

Stay another day for even more technical discussions at Google NY

In addition, I’m organizing a post-SMX technical day of follow up discussion, information, and site review time to really dig into what technical issues you are experiencing and help you formulate real solutions. This developer summit, as we’re calling it, is co-hosted by Jane and Robot and Google and will be held at the Google New York office on Thursday, October 8th, the day after SMX East. It’s absolutely free and SMX attendees get preference on invitations.

See the SMX East developer’s itinerary for complete information about what’s available for web developers and anyone else interested in the technical side of SEO and see the Jane and Robot developer summit page for more details on the agenda and information on how to register for your invitation. The technical highlights of SMX East are included below. I hope to see you there!

Day 1 – October 5

SMX Boot Camp: For web developers who are new to search and SEO, we recommend that you go through ALL the SMX Boot Camp sessions on the first day to get a solid foundation of how search works. If you’re more experienced, then…

Duplicate Content Issues: The Search Engine Edition – What’s the best way to deal with duplicate content issues? In this session, search engine representatives offer direct advice including tips on the new canonical tag.

Ranking Tactics For Local Search – This session looks at cutting edge tactics that work to drive traffic from local listings, plus highlights research to predict where things may be going.

Maps, Maps, Maps! – This session looks at how the mapping & local seach services work from the three major search engines. Search representatives will share insights and tips for performing better.

Day 2 – October 6

Ecommerce Search Marketing Tactics – This session looks at a variety of tips and tactics to help retailers and ecommerce merchants do well in search results.

Ask The Search Engines: Best Practices Edition – In this session, search engine engineers and representatives share their most important takeaways and tips for those hoping to do better in traffic, plus there’s lots of time devoted to open Q&A.

Ask The SEOs – PowerPoint Free! Put your questions about SEO to our panel of experts and get answers about ranking and crawling issues.

Universal & Blended Search Opportunities – This session looks at how to closely monitor your paid search visitors, analyze what they do when visiting your site and how best to convert them toward your goals.

Day 3 – October 7 – Technical SEO Track

Diagnosing Technical SEO Issues – Site isn’t being indexed fully? Experience a huge drop in rankings or traffic? This session provides a step by step checklist to help developers or others diagnose site issues and ensure they’re following best technical practices in SEO.

Dealing With Domain Names, URLs, Parameters & All That Jazz – Subdomains or subdirectories? Are tracking parameters a problem for search engines? Keywords in the URL make a difference? How about the “level” of a page – is “deep” in a site bad? This session focuses on technical SEO questions of this nature and provides answers.

Pumping Up WordPress For SEO – WordPress is an incredibly popular blogging platform and CMS system. This session shows you how to quickly tune WordPress with plugins and settings to get the most out of technical SEO.

CSS, AJAX, Web 2.0 & SEO – This session looks CSS, AJAX and Web 2.0 dynamic design techniques that can cause search engine indexing and ranking issues, with solutions to consider.

Register for SMX East and request your Jane and Robot Developer Summit invite!

A new feature has appeared in the Site Configuration Settings Sections of Google Webmaster Tools. The setting, called Parameter Handling, enables site owners to specify up to 15 parameters that Google should ignore when crawling and indexing the site.

Google lists the parameters they’ve found in the URLs on your site, and indicates whether or not they think they those parameters are extraneous (with a suggested “Ignore” or “Don’t ignore”. You can confirm or reject those suggestions and can add parameters that aren’t listed.

So what does this mean for site owners?

The primary value of the feature is to improve the canonicalization of a site in Google’s index due to duplicate content. Canonicalization issues occur when multiple URLs load the same content. This scenario can be problematic for a number of reasons (for instance, it can skew analytics data) but from a search perspective, canonicalization issues can cause:

• Crawl efficiency problems: if search engine bots crawl the same page via multiple URLs, they may not have resources to crawl as many unique pages on the site
• PageRank dilution that can lead to lowered search rankings: if external sites link to multiple versions of a page, each page has less Page Rank value than if all links were to one version
• Display and branding problems: search engines display only one version of the URL; you ideally want the canonical version of a URL to display (mysite.com/goldfish) rather than a version with extraneous parameters (mysite.com/goldfish?adid=1205123&sid=452006&sort=high-rating&loc=sea)

A number of canonicalization solutions exist, including several that are Google-specific, so why did they launch this new feature? Yahoo! has included a similar feature as part of its Site Explorer webmaster product for some time and site owners have been asking for a similar feature from Google for a while (certainly at least since I was working on Webmaster Central).

Below a rundown of the various canonicalization options and how this one differs.

Google Webmaster Tools Parameter Handling: When URLs Can Contain Optional Parameters

This new option only helps with canonicalization issues that are caused by optional parameters that are in a standard key-value pair format and that you specify. In other words, it can only be an exclusionary list (don’t crawl parameters x,y, and z) rather than inclusionary (only crawl parameters a and b).

Wouldn’t you always know the complete list of potential parameters? Hopefully. But some canonicalization issues happen because a URL can take any parameters at all. Ideally, you want to ensure your server isn’t set up this way, but if you need this configuration (for instance another team or outside agency needs the ability to use any custom tracking code without waiting for that parameter code to be added to the server set up), then you’re better off using the meta canonical tag.

The two most common reasons for optional parameters and that this feature will work well for are:

• Tracking codes used for analytics data (in this case, you may not want to implement a 301 redirect from the long version of the URL to the canonical one since you could lose the data)
• Page layout changes, such as sort orders (in this case, the code on the page uses the parameter to change the layout of the page, but from a search engine perspective the content on each version is the same, just in a different order)

Why use this canonicalization option over the others? The biggest benefit is likely in the increase in crawl efficiency. When Google discovers a new URL, they can check the included parameters against the parameter handling list and remove any optional ones before crawling it (but still credit any found links to the page). This could substantially reduce the crawling overhead on a site and could free up considerable bandwidth for getting other pages of the site crawled.

It’s also fairly simple to use. Just scan the list of suggested parameters and click the ones that are optional. In some organizations, it can be difficult to get source code added to web pages, making the implementation of the canonical tag difficult and time consuming. With this option, if you have verified webmaster tools access, you don’t need to involve IT at all.

What are the drawbacks to this option? The most obvious issue with this option is that it only works for Google. In time past, you could use this setting and the corresponding one in Yahoo! Site Explorer and not worry about other engines. But with Microsoft Bing’s impending (likely) replacement of Yahoo’s search index, it’s quite possible that Yahoo’s feature will go the way of its index, and if Microsoft doesn’t offer something similar, then a search index with 25%+ market share could be getting your URLs wrong.

You could also shoot yourself in the foot, metaphorically speaking. You could accidentally tell Google to ignore important parameters that, if dropped from the index, could wipe out large portions of your site. As Google adds more of these types of features to webmaster tools, it becomes more important to ensure that anyone who has access to them know what they’re doing.

In reality, Google likely has safeguards in place that at least partially protect against such accidental destruction. That’s undoubtedly why they say that “While Google takes suggestions into account, we don’t guarantee that we’ll follow them in every case.” They don’t want large portions of their index disappearing either.

Unlike accidental blocking with robots.txt, which search engines follow as a directive, this feature (and many of the others) is a signal only. If the other signals already in place strongly contradict it (for instance, the content seems to be vastly different), it likely won’t be used.

But even though Google has safeguards like this one in place, you may not want to chance it if you’re not confident of which parameters are really optional (all the time, since this is a site-wide setting).

This option also won’t work if your canonicalization issues aren’t related to parameters or if the parameters aren’t in standard key-value pair format.

Meta canonical attribute

The canonical attribute is a page-level meta tag that specifies the canonical version for the page. This can be useful because no matter what optional parameters are added to the version of the URL that renders the page, search engines can always know the canonical version. You can find detailed information about this tag in my article about its launch.

Why use this canonicalization option over the others? You just specify the canonical version of a page once, and no matter what parameters are added to the URL, search engines are always provided with the canonical version.

Since this meta data is on the page itself, any search engine can read it, and in fact, Google, Yahoo!, and Microsoft have all announced support for it. As of yet though, only Google seems to be actively using it.

What are the drawbacks to this option? Unlike with the parameter handling feature, search engines have to crawl the page before they can read the tag, so some crawl efficiency is lost. This tag should promote long-term efficiency, however, since theoretically,  once the bot has crawled the non-canonical version of the URL and read the tag, it shouldn’t have to crawl that version of the URL again.

As already noted, implementation requires modification of the page source code, which isn’t always easy within some organizations.

As with parameter handling, it’s possible to implement this tag incorrectly. For instance, it’s been discovered that some sites have accidentally set the canonical version of every page to the home page. As with the parameter handling feature, search engines consider the tag a “strong hint” as a precaution against these types of mistakes and won’t use the data when it strongly contradicts their other signals. In the case of Google, the only search engines who is actively using the tag so far, this has proven to be the case.

301 redirect

It’s  universally agreed that (other than not have multiple versions of a URL at all) the best way to canonicalize URLs is to redirect all versions to the canonical one using a 301 redirect. This implementation sends all users and search engines to the canonical version and effectively consolidates all links to the page and ensures only the canonical one is indexed and ranked.

Why use this canonicalization option over the others? It’s understood and followed by all major search engines and it provides the best user experience (visitors have one URL to access, bookmark, and share). In most cases, search engines consolidate all links to the redirect target and rank the canonical one.

This option is the best choice when you are moving content (for instance, changing your URL structure or changing domains) and to indicate whether your want content indexed under the www or non-www version of the domain.

Also keep in mind that if you redirect to the canonical version you’re more likely to get links to the right version, since most visitors will simply copy and paste what they see in the address bar.

What are the drawbacks to this option? When you are using parameters for sort orders or tracking, a redirect may negate those parameters. ou can generally configure your analytics program to handle this properly, but it probably won’t work out of the box.

Redirects aren’t always properly implemented. For instance, they might inadvertently be implemented as a 302 (or worse yet, a JavaScript redirect or meta refresh). Or they may generate redirect loops or infinite redirect chains. In these cases, search engine bots eventually abandon the crawl attempt (and with Google and Microsoft, you can get a list of these URLs in their webmaster tools products).

Redirects can also slow down crawl efficiency, particularly due to redirect chains. Ideally, search engines crawl the redirect then eventually stop crawling the origination URL, but if the bot encounters links to the original URL, it will continue crawling both versions (or more, if the page has moved multiple times).

Google webmaster tools change address feature

This feature enables you to tell Google when you’re changing domains. You have to verify ownership of both the old domain and the new domain and then you can specify a move from one to the other. You can find more information about this  feature xx.

Why use this canonicalization option over the others? The best use of this feature is when you are changing domains and you aren’t able to implement a 301 redirect from the old domain to the new. (This is the case, for instance, with blogspot.com sites.) Even if you are able to implement the redirect, it can’t hurt to let Google know as well!

What are the drawbacks to this option? You can only use this option to move from one domain to the other. And as with the other Google webmaster tools features, it only works for Google.

Google webmaster tools preferred domain feature

The preferred domain feature enables you to tell Google whether your want your domain indexed with the www subdomain or without it.. Since most sites resolve either way, a complete duplicate set of content  of your site will exist if you don’t set www/non-www canonicalization. Why is this a problem? Ideally it’s not and search engines consolidate the content correctly. But often, search engines find links to both versions and end up crawling both, indexing both, and crediting the links to the versions separately.

Why use this canonicalization option over the others? You may as well always use this option, although you should implement a 301 redirect as well, if you can. Google initially implemented this feature for those sites that weren’t able to do so.

What are the drawbacks to this option? Again, this option works only for Google. And it doesn’t provide as much of a guarantee as a 301 redirect.

Blocking duplicate content with a robots directive

The traditional advice for avoiding duplicate content has been to block the duplicates with robots.txt (or a robots meta tag) to ensure the correct version is indexed. It can be important that the right version be indexed vs. the version intended for print, for instance.

Why use this canonicalization option over the others? Generally speaking, you shouldn’t now that the canonical meta tag is available. The scenarios for which you wouldn’t want to redirect (such as the print version example) can be more easily solved with the canonical tag and the scenarios for which you’re worried about crawl efficiency issues that would leave large portions of your site uncrawled (such as large-scale optional parameters) can now more easily be solved with Google’s parameter handling feature.

What are the drawbacks to this option? The primary drawback to this option is the loss of link credit. Any links to blocked pages fall into a black hole and can’t be credited to the canonical version of the page, as happens with the other options.

The parameter handling feature can also provide insight on how Google sees your site

For some time, Google has been attempting to canonicalize URLs and show the canonical version in the results, even when a site owner hasn’t implemented any of these canonicalization options. For instance, they may determine that several pages contain the same content and algorithmically consolidate them and associate them with the one Google determines is canonical. They haven’t described exactly how they determine the canonical version, but they might, for instance, choose the URL with the fewest number of parameters or the shortest version of the URL.

Last year, they started letting webmasters know when they encountered URLs that they thought were extraneous and were causing crawling problems. It’s likely that Google is using a similar source to generate the list of parameters it suggests should be ignored.

In this way, the parameter handling feature provides insight into how Google perceives the site. If you see many parameters listed that aren’t optional, take a look at the content on the URL that use those parameters.

This could signify a larger problem. It could be that Google doesn’t see enough unique content on them (this can happen, for instance, with pages that list part numbers, contain mostly images and item codes, or list little information outside of a login). You may want to look for ways to differentiate the pages a bit more.

Interestingly, the Google Webmaster Central blog has a new post about duplicate content, but no mention of this new feature. Thanks to Brian Ussery for pointing it out!

Google’s August 29th logo notes Michael Jackson’s birthday.

Google has commemorated birthdays of many famous people and events, such as Luciano Pavarotti, Lego, and Dr. Seuss.

Google has just unveiled a “secret project” of “next-generation architecture for Google’s web search“. This new architecture appears to include crawling, indexing, and ranking changes. For the first time, Google isn’t simply incorporating these changes into their existing infrastructure or replacing it. Instead, they’re providing a developer preview and are asking webmasters and power searchers to try it out and give them feedback. Unlike Google’s now-defunct SearchMash, which was intended for search experiments that wouldn’t necessarily be incorporated into Google’s main web search, the caffeine index seems to be an entirely new search infrastructure that will repace what exists now.

Based on the blog post, we can guess that this new infrastructure may include ways of crawling the web more comprehensively, determining reputation and authority (possibly beyond the link graph and what’s typically thought of as PageRank), and returning more relevant results more quickly, although Google’s Matt Cutts told me that the changes are “primarily in how we index”.

While the biggest visible changes in Microsoft’s relaunched search engine, Bing, are user-interface related, Google’s new search is only infrastructure related and includes no UI changes. On first glance, however, the underlying infrastructure changes do seem to have impacted user interface as it relates to universal search (likely because universal results are influenced by ranking and relevance signals). For the sample searches I did, the first ten results were fairly similar, but the existence and location of images, video, news, and blog posts was notably different.

A search for [buffy the vampire slayer] on the new infrastructure, for instance, returns video and news results midway down the page.

A search on the existing infrastructure, however, returns news at the top, video in the middle, and images at the bottom of the page.

Undoubtedly, Google Caffeine will cause quite a kerfluffle in the web developer and search engine optimization world and many will dive in to try and figure out the changes. We’ll likely see many a speculative blog post about how best to optimize for the new infrastructure, but my guess is that it likely does what Google search does, but better. And the foundational elements of having a crawlable infrastructure and compelling content remain.

The “Kay Bailey Hutchinson for Governor Official Website” put up by “Texans for Kay Bailey Hutchison, Allan Shivers, Jr., Treasurer” over the weekend in support of Kay Bailey Hutchinson for Texas governor was briefly in Google but now appears to be completely missing. Huh.

Odd since Bing seems to have indexed it just fine, although the snippet looks a little odd.

Let’s take a closer look at the site.

Looks OK. Maybe it’s all in images or JavaScript or something and Google is having a hard time extracting content? I’ll just take a closer look with web developer toolbar. I’ll disable CSS and…

Oh. Wait.

Do you think all those hidden keywords revealed when CSS styling is disabled might have anything to do with it? I speculated yes when first writing this piece, and after it was posted, I got this confirmation from Matt Cutts, head of the spam team at Google:

Google did take action on this site for hidden text. Hidden text is a violation of our quality guidelines. We’ve removed the site from our index and tried to contact the site maintainers by email to explain that the hidden text was the cause for the site’s removal from our index. We also recommended that the webmaster remove all the hidden text and file a valid reconsideration request. More information about requesting reconsideration of a site can be found here.

Hiring an SEO firm is just like hiring any other service for expertise that you lack. Since you by definition don’t have the expertise, how do you know if they’re doing a good job? Well, Texans for Kay Bailey Hutchinson, Allan Shivers, Jr., Treasurer, today you’re getting a free crash course in SEO evaluation.

Understanding hidden text

Hidden text is not only against the search engine guidelines and can get you banned, but is amateur and lazy. It’s like hiring a painter to paint your house and having them show up and throw a bucket of paint in the general direction of your walls.

You might find Google’s webmaster guidelines on the subject helpful:

Hiding text or links in your content can cause your site to be perceived as untrustworthy since it presents information to search engines differently than to visitors. Text (such as excessive keywords) can be hidden in several ways, including … Using CSS to hide text… If your site is perceived to contain hidden text and links that are deceptive in intent, your site may be removed from the Google index, and will not appear in search results pages. When evaluating your site to see if it includes hidden text or links, look for anything that’s not easily viewable by visitors of your site. Are any text or links there solely for search engines rather than visitors? If you do find hidden text or links on your site, either remove them or, if they are relevant for your site’s visitors, make them easily viewable. If your site has been removed from our search results, review our webmaster guidelines for more information. Once you’ve made your changes and are confident that your site no longer violates our guidelines, submit your site for reconsideration.

It’s quite possible Google has even told you about this hidden text problem. Create a Google Webmaster Tools account and verify ownership of standbykay.com. You may have a message waiting for you.

Hidden text is also frustrating for the users you’re trying to attract and may cause them to get irritated at your site and leave. Let’s say, for instance, the site does indeed rank for “what is a keg”, as it apparently is trying to do. What exactly do they expect searchers who reach the site to do? The site doesn’t actually answer the question of what a keg is. Do they think that all these keg definition-seekers are going to get distracted and decide to abandon their beer research efforts and become political activists?What other searches are they apparently targeting?

• texas bbq pit texas smokehouse
• chicken fried steak recipe
• debachary [sic] definition
• gambling addiction
• houston rockets
• why do we have knees

Why do we have knees?!!!

According to the Austin American-Statesman, which alerted us to this situation, the campaign has no intention of the removing the hidden text:

Hutchison’s campaign spokesman, Jeff ] Sadosky and other campaign aides said this afternoon that only the two phrases using “rick perry gay” will be removed because they won’t play into the campaign’s future messages.

[Note: At 2:22pm Eastern, it appears all of the hidden text is now gone.]

The explanation is that software is creating the lists of phrases based on search volume and is intended to help direct online advertising efforts. If that’s indeed the case, wouldn’t the list be more easily managed as an offline report than as text hidden inside source code of a web page? Just saying. [Can you tell I'm not buying it?]

Which brings us to the next lesson in the SEO crash course.

Understanding keyword research

Sure, there are sophisticated and expensive keyword research tools that provides lots of awesome information, but I’m venturing a guess that you’re more in the beginner stage, so you’ll do just fine with the free stuff. And as a bonus, the free tools give you files you can open in Excel rather than store the data in the source code of your site! The campaign said the software helped them figure out what else those who were searching for “Rick Perry,” “Kay Bailey Hutchison” and “Texas” were looking for, so let’s do the same, shall we?

Google AdWords Keyword Tool - this is the old standby, and it’s pretty solid and useful since it’s based on Google searches. And hey, it even lets your sort by search volume!

Tip: “Texas” might be overly broad as you’ll likely get a lot of untargeted traffic. You might want to stick with more relevant keywords, such as “texas politics” or “texas governor race”. A more targeted set of keywords can give you lots of other valuable information, like that opposing candidate has more searcher interest:

Google Insights For Search -This cool tools lets you compare search terms and zero in on specific regions (such as just Texas) and specific time frames (such as the last 90 days).

Wordtracker Keyword Questions – this new tools uses ISP data to generate a list of questions people are searching for related to keywords. Using this tool, we find that perhaps Texans for Kay Bailey Hutchinson, Allan Shivers, Jr., Treasurer has been doing a few searches of their own.

Here’s another tip! You can use these keywords in the actual copy of the website. It can help you not only be more visible in search engines, but also have deeper engagement with your consituents! How awesome is that. If you want to know what people are searching for, why only use that for ad targeting, when you can use it to understand and reach your audience more generally? For instance, people are apparently asking the question “why is Kay Bailey Hutchison running against Rick Perry for governor of Texas?” Add a page to the site with frequently asked questions, add that one, and answer it.

Understanding basic SEO

Texans for Kay Bailey Hutchinson, Allan Shivers, Jr., Treasurer, there’s another problem with your SEO. Not only does the site have elements that will get it banned, the site doesn’t have thebasic stuff that can help you be found. Let’s run down just a few of them.

Extractable text- None of the visible text on the page is viewable by search engines! (Kinda funny, huh. None of the visible text is viewable, but all the hidden text is.)  It’s all hidden in JavaScript. With images and JavaScript turned off, the page looks like this:

I should clarify, it’s not that none of the text is visible. “Political ad Paid for by Texans for Kay Bailey Hutchison, Allan Shivers, Jr., Treasurer” shows up just fine. So anyone looking for political ads paid for by (etc.), just might find the site. Well, if it wasn’t banned by Google for hidden text, of course.

Meta tags – The keywords, robot, and revisit-after tag are completely ignored in this context. So I hope you didn’t pay very much for someone to include them. The description tag is fairly important as it’s your one and only chance to provide a compelling marketing message in the search results to motivate a searcher to click over to the site. Was “Kay Bailey Hutchison – Running for Texas Governor” the most inspiring message you could come up with? Video – Don’t autoplay video when the page loads. That’s not really an SEO thing. Just an annoying-the-hell-out-of-me thing. More to the point, add at least a caption above or below the video. Add some descriptive text. Something that lets the search engines know what it is.

Canonicalization - it sounds like an advanced SEO tactic, but it’s actually a foundational element that even the most novice SEO should know. www.standbykay.com and standbykay.com both resolve to the site. One should 301 redirect to the other.

Duplicate content – Texans for Kay Bailey Hutchison, Allan Shivers, Jr., Treasurer also runs another site, texans.forkay.com. (Why the subdomain? No idea.) They also own texansforkay.com, which has a JavaScript redirect to texans.forkay.com. First lesson in redirects: always go with a 301 redirect.

The bigger issue is why two sites that have exactly the same purpose? So far, standbykay.com doesn’t have much content (and only one page), but the bulk of it is that dang autoplaying video, which is the same video featured prominently on the texans.forkay.com home page.

That leads me to think that any expansion of standforkay.com might use other content from texans.forkay.com. That’s another way to ensure the new site won’t rank in search engines, as they don’t want to list the same content multiple times. Any good SEO firm should ask why you need another site and work closely with you to define differentiation — a different purpose, audience, and goals.

But wait? What’s this? kayfortexas.com? That appears to be an exact duplicate of texans.forkay.com. Studying up on that 301 redirect is probably a really good idea.

I hope Texans for Kay Bailey Hutchison, Allan Shivers, Jr., Treasurer enjoyed this crash course in SEO. If you found it helpful and are want a second lesson, might I suggest Google’s informative Requesting reconsideration of your site.

By now, everyone has read all about the news that Yahoo is replacing its search index with Microsoft’s Bing. In a way, it’s a great story of complete reversal, as in 2002, Microsoft didn’t have its own index and instead used Inktomi. Late that year, Yahoo! acquired Inktomi, which spurred Microsoft to start building its own search index to avoid having a search supplier owned by a major competitor. Now Yahoo is ditching its index (including all of the technology it acquired with Inktomi) to use the very index it motivated Microsoft to build.

We’ve read about what this means for advertisers (more overall traffic from the combined audience, use of Microsoft adCenter for self-serve and Yahoo!’s sales force for premium) and for searchers (they likely won’t notice), but what does the deal means for those who create websites: publishers, web developers, and SEOs?

Web Developers

The hardest hit by this change will likely be developers. Over the last couple of years, Yahoo seems to have shifted its focus from innovating the search index to innovating its developer offerings: encouraging third-party development and creating a “developer ecosystem” for search.

Any developer options that don’t rely on the Yahoo search index may be unscathed. In particular, the non-search development tools and search-related offerings that are solely focused on the user interface may continue to be supported. While Bing will power Yahoo’s search engine, Yahoo will control their user interface and likely will try to continue to differentiate there. That’s means Search Monkey, which enables site owners to enhance how their results appear on Yahoo, is potentially safe.

Build Your Own Search Service (BOSS) likely won’t be so lucky. BOSS is built on the Yahoo index as its foundation. A company can build their own search engine using Yahoo’s underlying technology and differentiate via the user experience.  Essentially, that’s what Yahoo is planning to do now with Bing as their underlying technology. No more Yahoo index likely means no more BOSS. Yahoo all but concedes as much: “We can tell you that BOSS will remain live for the time being.” What does that mean for companies like hakia, OneRiot, Daylife, and Cluuz? And for that matter, all of the developers using BOSS who are now filling the Yahoo BOSS message boards with questions?

On the Yahoo developer blog, Yahoo commented that “For SearchMonkey and BOSS, we currently do not have anything concrete to tell you. Clearly, we’ll need to work with Microsoft to determine what makes the most sense for you and for us.” If BOSS’s future is left up to Microsoft, I have no doubt that future will involve migrating BOSS users to the Bing search API. In order to continue to support BOSS, Microsoft would have to completely recreate it to work with the Bing search infrastructure. Why would they do that when they can increase the audience of a product they already have? It’s possible they’ll add some of the unique BOSS features their search API (such as unlimited queries, ability to mash up the data with other sources, and ability to tweak ranking signals), but I wouldn’t hold my breath. The Yahoo BOSS team is just as in the dark as the developers wanting answers. From a message board post: “What specifically does it mean for BOSS? Honestly the team is still absorbing the implications and we just don’t know.”.

BOSS users could switch to Google’s Custom Search API, but it is more restrictive than Microsoft’s offering, and isn’t really well-suited as the foundation of a search engines or other commercial company. Several other companies offer web indices, such as CommonCrawl and Alexa, so perhaps they or a new company will take advantage Boss’s imminent demise and offer matching features.

Any Yahoo offerings that don’t rely on an underlying index, such as the Yahoo User Interface library are likely going to remain. Yahoo confirmed this in their blog post:

“We’ve also received questions about the future of Yahoo!’s other developer offerings, such as YUI, YQL , and Pipes. We wanted to let you know that today’s news does not affect these products. None of our other non-search developer products are affected.” [Emphasis mine.]

However, look for any search index-based offerings (such as the Maps API and Local API) to be deprecated in favor of the Bing equivalents once the deal goes through.

Search Engine Optimizers and Site Owners

What about those who are concerned with getting customer acquisitions through organic search? How will this change impact them? From a traffic perspective, take a look at how well you’re indexed and ranked in Bing. That’s how well you’ll be indexed and ranked in Yahoo. What do your titles and descriptions look like in the results in Bing? That’s how things will generally look in Yahoo. This might not be a bad thing for site owners, as over the last year, Yahoo’s search quality seems to have been declining to the point that I’ve been wondering if their engineering team had already begun to be phased out or least was spending a lot of time at the bar mourning the likely phase out.

Just as you don’t need to optimize separately for AOL since they use Google’s index, you won’t need to optimize for Yahoo since they’ll use Bing’s index. The exception to this may be in how Yahoo displays results. We’ll have to wait and see exactly what this means, but Bing has been trying to differentiate in display and it supposedly, Yahoo will continue to do that as well. This may mean, for instance:

• SearchMonkey will continue to be important as a way to stand out in the results.
• Hmm. I can’t really think of anything else.

My guess is that the robots-nocontent tag will no longer be supported, since Bing’s infrastructure doesn’t support it. The search engines have already come together to standardize their support of robots.txt and XML Sitemaps, so site owners shouldn’t worry about changing anything with those.

The bigger issue many SEOs are concerned about is Site Explorer. Site Explorer is one of the more reliable tools for competitive backlink research. You can see a substantial list of links to any site, generally ordered according to value. That’s useful stuff! Both Google and Bing Webmaster Tools provide backlink data, but only for your own sites. Yahoo will be unable to maintain Site Explorer without a search index of their own. Will Bing take it over? Well, it could add the feature to its Webmaster Tools, but Microsoft has historically been moving the other direction. They removed the ability to query their index for link data with the link: operator in 2007 and have never brought it back for competitive research.

Microsoft likely won’t be motivated to add a feature that they specifically chose to remove. And it’s not trivial to build the code to query for competitive links and store the data. Believe me, I know. I managed the process for adding non-competitive backlink data to Google Webmaster Tools. As with the potential end to BOSS, the potential end to Site Explorer opens up new opportunities for third-parties. In fact, the same companies who build a web index could provide competitive link data. Currently, SEOmoz  provides Linkscape, which offers some similar features. (Speaking of SEOmoz, Rand Fishkin posted yesterday about the SEO impact of this deal.) Majestic SEO and Exalead provides link data as well.

More generally, will Microsoft step up its efforts with webmaster relationships? Yahoo used to have a fairly significant presence in the community. In addition to Site Explorer, they were a constant at conferences and participated in online discussions. That participation has declined lately, coinciding with the decline in search quality. Microsoft seemed to be rallying with its webmaster relationships with the launch of the Webmaster Center in November 2007. But Microsoft hasn’t updated the Webmaster Center with new features since August 2008. (A minor release in November didn’t add  new features).

Microsoft didn’t respond to my questions about their current and future resource investment in this area. They did recently release a rudimentary SEO Toolkit, although it requires Windows Vista and IIS 7.0 to run.

And what about paid inclusion? Yahoo has long offered Search Submit Pro, which essentially enables sites to pay to be included in the organic listings. Microsoft doesn’t offer a similar product and while it’s certainly possible that Microsoft will add this product to their offerings, paid inclusion is quite a substantial shift in overall approach to organic search. It’s less about the ability to implement the technology and more about belief around what constitutes an “organic” index. Danny Sullivan asked about paid inclusion at the announcement press conference. Carol Bartz, Yahoo CEO replied, ” Paid inclusion, we’ll decide on that later.” But it would be difficult for Yahoo to continue the program on its own, as Yahoo will no longer have control over what pages are included in the search index.

In the end, it’s about the traffic

The big question is will this partnership significantly change market share percentages? Depending on whose numbers you use, Google has either 65% or 74% share in the US (more in some European countries). That puts the combined Yahoo/Microsoft share at 28% or 25.5%. That’s substantial traffic, sure, and worth paying attention to. But what will the share look like in three years once the deal is done and we barely remember Yahoo ever had its own index? My guess is pretty similar to how it looks now. Except Google will probably have slightly higher share. I just don’t see anything game changing here that will cause a mass exodus from the status quo. But I’ve been wrong before. What I do know is that site owners who have ignored how their sites were doing in Bing until now do to low traffic numbers will likely start paying a lot more attention.

Yahoo started the push to structure the web when it launched SearchMonkey. With SearchMonkey, a web developer could add semantic markup to a page and then build an enhanced listing for Yahoo search results.  Not long after, they made things even easier by using this data to enhance listings even without a SearchMonkey application. Google jumped on the structured data bandwagon as well, picking up things like reviews information to provide additional information in search results. Microsoft Bing’s new document for webmasters also encourages the use of structured data, although it’s not clear what they do with the data. Yahoo continued its structured data campaign last week, with the announcement of Common Tag, a vocabulary that extends RDFa.

Now Yahoo has announced support for several additional opportunties for automated enhanced listings.  You can use DataRSS XML to send a feed directly to Yahoo or use RDFa or microformat markup on your pages for content such as products, events, and news. In addition, Yahoo can now use a Google Base feed to create enhanced listings. Simply submit your Google Base feed through Site Explorer. As I mentioned in my analysis of Common Tag, these moves seem to be motivated at least in part by Yahoo’s quest to increase the reach of BOSS. Yahoo’s blog post notes that when you submit your Google Base feed, Yahoo converts it to DataRSS and makes it available in BOSS so it can be used for third-party search engine development.

While none of the search engines are using structured data for crawling, indexing, or ranking, it’s still likely worthwhile to include this markup for data like events, news, and reviews to take advantage of the opportunity to have a search result that stands out in Yahoo and Google (and potentially in Bing as well).

Microsoft recently published a PDF about Search Engine Optimization called Bing: New Features Relevant to Webmasters. This is the second SEO-related offering in as many weeks. During SMX Advanced, Microsoft launched the IIS Search Engine Optimization Toolkit. I’m working on a review of that as well, but it requires Vista or Windows 7 and IIS 7, so installation takes a bit of effort. I talked with Microsoft about the information in the PDF and got some clarity and additional information. The PDF primarily describes the user interface changes launched with Bing, and how those might impact site owners, but also touches on search engine optimization.

Google has provided SEO advice for a while, both in their help center and in their Search Engine Optimization Starter Guide, so check those out if you’re interested in hearing the official search engine stance on SEO. Google’s document is much more instructive, providing their top guidelines for creating websites that can be easily crawled and indexed by search engines.

What can we learn from Microsoft’s advice for webmasters?

The document starts by describing the primary difference in Bing vs. Live Search that impacts site owners: the first search results page no longer shows results 6 through 10 for the searcher’s query. Instead, the page includes “categorized” results after the list of 5 sites that match the initial query. So, for instance, if you search for [vw beetle], results 6 through 8 are for [volkswagon bettle for sale] and results 9 through 12 are for [used volkswagon beetle].

Expanded long tail opportunities

The document clarifies that more than 300 factors are used in determining this categorization and that what categories appear is entity-dependent. In other words, if you search for [buffy the vampire slayer], you may see a category for “wallpaper” because lots of people search for Buffy wallpaper, but you might see categories for “episodes” and “characters” because Microsoft has identified “Buffy the Vampire Slayer” as a TV show.

Microsoft says that this new way of organizing results provides new long tail opportunities for site owners. They suggest determining what entities you most want your site to rank for and then optimizing for the long tail queries within those entities. For instance, if your site is about cars, you’ll want to optimize for the car brands (VW, Volvo, etc.) but also car-related categories such as dealers, used, and for sale. That will give you additional opportunities to rank on the first page of results.

Best match

Bing has also introduced a concept called “best match”. When Bing is reasonably confident that the first result is what the searcher is looking for, it will be noted as the “best result” and could contain additional details, such as a box for searching within the site, phone numbers, and links to pages within the site. In some instance, no results other than the best match appear on the page, which Microsoft says their team chooses when they have high confidence in the result and the query volume is high. Of course, this isn’t great news for the site owner at position 2.

If you don’t want your site to appear as the best match, don’t want the internal site search surfaced, want to correct the phone number that appears, or want to remove links, you can request that by emailing bestmatc@microsoft.com.

Document preview

Document preview provides additional content from the site in a hover and, in Microsoft’s words, “helps searchers find the content they want faster, without leaving the SERP until they are ready”.

They say this helps increase qualified traffic, but some webmasters might think that it helps searchers find the content they want without leaving the SERP at all. For them, Microsoft provides a way to opt-out of the feature. Just add the following to the <head> section of each page:

<meta name=“msnbot”, content=“nopreview”>

If you don’t want this feature used on any page of the site, it might be easier to return the directive in the server HTTP header as follows:

x-robots-tag: nopreview

Not everything from the document preview comes from the site. Some information could be from external sources as well. For instance, if the site is in Flash and Microsoft has trouble extracting data, they might turn to a third-party source. And they may use local information such as an address or phone number from an external source. You can request that third-party data not be used for your listing by emailing bestmatc@microsoft.com.

Flash extraction

Microsoft says that Flash-based sites are responsible for 21% of all empty descriptions in their index. They say they are doing “limited data extraction” and are now able to generate descriptions for one-third of those. They may also use anchor text from incoming links. Of course, Google has been using incoming anchor text as source data for missing titles for a while, and has continued to evolve its ability to extract Flash data. No word on whether Microsoft is using Adobe’s crawler API or another technology for this extraction.

Use of microformats

While the document encourages the use of structured data, such as microformats, Microsoft tells me that they aren’t currently using this data for crawling, indexing, or ranking. This stance mirrors Google’s and Yahoo’s, both of whom are encouraging structured data use, but are not yet using the data for web search.

Instant answers

Instant answers, the “OneBox”-style results that provide data to answer the query, aren’t new to Bing, but may be gaining prevalence. For now, Microsoft tells me that you can request evaluation of your data for inclusion using their support forum.

Local listings

Local results are more heavily featured in Bing. You can add your site to their Local Listing Center to ensure you have the opportunity to be featured in these results.

Searcher behavior

Microsoft provides some interesting data about searcher behavior, which they tell me is based on a number of internal studies derived from analysis of their search logs and toolbar logs. For instance, they found that searchers refine their query, bounce back to the search result, or abandon the search 50% of the time. The breakdown of that 50% is below.

They also found that searchers repeated 24% of their queries during a session (which led to the additional of search history in the left column of the search results page).

All in all, this document doesn’t provide a lot of new information about SEO. But I applaud Microsoft for understanding things like removing results 6 through 10 and replacing them with categorized results and adding a hover with additional content from the site could impact search traffic and for providing information about these features. It’s difficult to say just how much these changes will effect traffic patterns. The biggest change, of course, would come if Bing brings about Microsoft’s goal of increasing search share.