Google explained, “instead of only focusing on quick answers to specific questions, we’ve created some longer videos which cover important webmaster-related topics.” The first video answers a question on if there is a limit on the number of 301 redirects you can have.

Matt explains that there is no limit in the number of 301 redirects from a site to another site. For example, if you are migrating tens of thousands of pages from one site to another, you can set up 301 redirects for all of the pages. But there is a limit of about 4 or 5 hops in a redirect. For example, you have page A that redirects to Page B that redirects to Page C and then to Page D, it might not follow up to or including Page D.

To date, Google has over 375 videos on their YouTube Google Webmaster Help Channel. Currently, most around about a minute or so long, but the more detailed versions are coming soon.

Related Stories:

• Google’s Matt Cutts Live Q&A On YouTube Now
• Google’s Matt Cutts On “Over Optimization”
• Google Webmaster Video Reconfirms Use Of Social Signals
• Google’s Matt Cutts On Keywords In The URL
• Matt Cutts Webmaster Video Infographic

Both James and I sent this information to Google as soon as we heard of it. After several hours, Google has told us, “we’re still investigating this report, and to be cautious we disabled all URL removals earlier this morning.” So now, if you even own a site, you won’t be able to remove the site or pages from the site using Google’s URL removal tool.

How did this loophole work? Pretty simple as James described. You use the following URL when logged into Google Webmaster Tools:

https://www.google.com/webmasters/tools/removals-request?hl=en&siteUrl=http://{YOUR_URL}/&urlt={URL_TO_BLOCK}

Then replace {YOUR_URL} with a URL you control within Webmaster Tools, and replace {URL_TO_BLOCK} with the URL of the site you want to block.

You could block a whole site, section or single page this way, based on how you entered the URL. To block a site, use the top level domain (E.g. http://www.someurl.com/), to block a section (subfolder) use a subfolder URL (E.g. http://www.someurl.com/somefolder/) and to block a page use the specific page URL (E.g. http://www.someurl.com/somefolder/somepage.html).

I am waiting an update from Google on why this happened, if site’s were impacted and how long this was an issue.

Postscript:: Google sent us a statement that they have fixed the issue. A Google spokesperson said:

We’ve confirmed that there was an issue within the URL removal feature in our Webmaster Tools and have already pushed out a fix and re-enabled URL removals.

The URL removal feature keeps detailed records, so we’re currently reprocessing earlier removal requests to ensure their validity. Our initial examination has shown only a limited impact.

This enables webmasters to set up a canonical for linking to or from PDF files or other non-HTML based files.

One of the examples given by Google is that a “webmaster can signal to Google that the canonical URL for the PDF download is the HTML document by using a rel=”canonical” HTTP header when the PDF file is requested.”

This also comes in handy when you use a CDN (content delivery network) and the content is being served from many different URLs. You can now use the rel=”canonical” within the HTTP header to set up those redirects for that type of content.

Related Stories:

• Google, Yahoo & Microsoft Unite On “Canonical Tag” To Reduce Duplicate Content Clutter
• Canonical Tag 2.0: Google To Add Cross Domain Support
• Google Supports Cross-Domain ‘Canonical Tag’

I’ve pulled out the key points from this post in bullet format:

• Google differentiates between traditional mobile phones and smartphones.
• Google has two bots: Googlebot and Googlebot-Mobile. Googlebot crawls desktop-browser type of webpages and content embedded in them and Googlebot-Mobile crawls mobile content.
• Currently only traditional phones are supported with special useragent strings within Googlebot-Mobile, not smartphones (this may change)
• Google said they “expect smartphones to handle desktop experience content so there is no real need for mobile-specific effort from webmasters.
• It does not mean you can’t serve a special style sheet to smartphones, Google said, “the decision to do so should be based on how you can best serve your users.”
• URL structure: For Googlebot and Googlebot-Mobile, it does not matter what the URL structure is as long as it returns exactly what a user sees too.
• Using the same URL “is not considered cloaking by Google.”
• Mobile sitemaps: you should include only mobile content URLs in Mobile Sitemaps, even if these URLs also return non-mobile content when accessed by a non-mobile User-agent.

This blog post was in response to a question I had in regards to a video Matt published, which is below. But make sure to understand that you do not need to use a mobile specific URL, you can serve up mobile content on the same URLs you serve up to desktop users. In addition, it is important to note that smartphones like iPhones and Android phones are not necessarily and currently considered “mobile” devices to Google or Googlebot-Mobile.

Related Stories:

• The New Mobile SEO: What You Need To Know
• Two New Mobile SEO Tips For 2011
• Has The iPhone Made Mobile SEO Obsolete?
• Mobile SEO Tip: Transcoding Services Can Dilute Link Popularity
• Don’t Penalize Yourself: Mobile Sites Are Not Duplicate Content
• Why Mobile Searchers Need Mobile-Optimized Sites
• Sorting Out The Mobile Search & SEO Mess
• Google Writes On Mobile Site SEO Concerns & Techniques

Here’s a weird one. Try a search for the BBC TV series Doctor Who on Google. You’ll find the official site, but look closely at the domain name. It’s amirgabriel.com, as shown above — not bbc.co.uk as it should be.

What’s up with that? Has the BBC undergone a transformation in the way Doctor Who himself undergoes?

It’s not a case of someone redirecting their domain over to the BBC and somehow “hijacking” the listing, a problem more common a few years ago. That’s because the domain doesn’t do a redirection, take you from one place over to the BBC. Instead, it “resolves” directly to the BBC –there’s no forwarding at all.

Maybe it’s a BBC-owned domain that’s used as an alternative for the BBC site for some reason, though it’s hard to tell if the BBC owns it because the “whois” owner information is protected.

However it happened, Google thinks bbc.co.uk and amirgabriel.com are the same, as this search shows:

While it’s not uncommon for a web site to have multiple domain names, usually Google will display the most popular or most known one in its listings. That’s not happening in this case, and it should. Even Google agrees. The company told me:

This is a known bug and we’re working to address it

If you’ve gone to a sports bar for lunch lately, you’ve undoubtedly read some pretty funny closed captioning on the live news or sports channels running in the background. I can just imagine these poor guys typing at a frenzied pace, trying feverishly to keep up with everything being said. Your web analytics people aren’t so different when they’re trying to record the whole show unfolding on your site, and at times you probably wonder if they caught everything.

What causes issues?

Analytics implementations are often slapped together pretty hastily. One of the things that your analytics implementation person can struggle with is keeping track of new content and making sure that analytics is properly set up on new pages. Web analytics people aren’t usually as involved as they should be in the content creation process, so when they’re asked to make sure “it all works,” they often have to scramble to get the basics in place, hoping they will have time to set things up the way they really should be. And if you’re a paid search marketer creating great landing pages or an SEO working on developing a whole buffet of new content, please get the analytics guys involved early. Who knows? They may even have some good ideas or be able to shine some light on history that will help you out.

I’ve probably worked on about 100+ implementations of both Omniture SiteCatalyst and Google Analytics, and I have literally seen only one single, solitary good implementation… ever. And, unfortunately, I usually find out that there were a number of real-world issues that lead to that point (it would actually be reassuring to know that the implementer just didn’t know what they were doing; it’s always much harder to know that competent people couldn’t succeed because of crummy circumstances). So, let’s try to do what we can to help the analytics peeps out, because at the end of the day (I still really hate it when people say that), we really rely on them to illuminate the effect of this PPC and SEO work.

Beyond creating new content, you may be implementing redirects to support an SEO effort and improve your URLs or structure (or creating vanity URLs for other campaigns, etc.), and there is a lot of confusion around how analytics tools will react to these changes. I wrote a comprehensive guide to how Google Analytics and Omniture SiteCatalyst treat 301 (and 302) redirects, so if that’s something you’ve been trying to work out, please go take a look and fire off any questions you may have.

Redirects are especially touchy for a few reasons, two of which are pretty important:

1. They can cause issues with traffic sources: if your paid campaign parameters aren’t retained post-redirect, your referral data may have some major issues. For one thing, your paid search traffic may be getting attributed to natural search! Yikes. Read that other post for more details.
2. If you have 302 redirects in place, or 301s that have not yet been digested by the engines (the engines may still link to the old pages; new ones are not yet indexed/in SERPs), you may be really confused to see some landing pages for some keywords. If you are comparing the URLs in the SERPs (or a ranking report) to the landing page URLs you’re seeing in your analytics tool, these babies are not going to line up, and may give you a false impression of which pages are generating traffic for which keywords.

Some of the documentation from Omniture and Google is a little misleading or outdated when it comes to redirects, so be careful. Google has one misleading help page, while Omniture has one great page and two very misleading, contradictory pages—I can’t link them because you’ll need credentials to view them. Unfortunately, this confuses a lot of people. If you are an Omniture client and want to discuss it in more detail, send me a reply on twitter (@evanlapointe) and we’ll see if we can get things figured out (and read my other post first for some tips that might help you diagnose). Also, Avinash Kaushik covers these issues very well in his books, and the dudes at Omniture who are on twitter seem to be super sharp guys if you want to talk to them directly. Hopefully, they’ll find this post and put their info in the comments.

Some things you can do to make sure you’re getting the credit you deserve:

Put your analytics person on speed dial. It’s going to be helpful for you to know who is responsible for analytics implementation in your organization, and if you have a huge site, who else might be responsible for implementation in other site sections, subdomains, third-party shopping carts, sites recorded under the same suite/profile, etc. Talk through some of the issues you’ve seen with tracking in the past. And try to get friendly enough to where this person calls you once a week or so to give you some good news or ammunition for your boss or next meeting. It’ll be worth it.

If your analytics tool allows them, use annotations as much as humanly possible. Mark dates that redirects go up. Mark dates when products or content go live. Mark anything you think will ever be remotely important, because when your data spikes, everyone’s going to go running around to figure out why. This will save everyone a lot of time and energy.

Collaborate. Sit down with the usability, design, copy and IT people, along with analytics (or if you’re responsible for all of these things for your company, have a nice cup of coffee and talk to yourself for a while) and discuss what you really want to accomplish past simple traffic generation. What do you want people to do first? Then what? And then? Get micro. Talk about how you can all learn and iterate based on what you learn. Get on the same page and who knows what’ll happen.

A common practice in mobile web site design today is to redirect a mobile user to another domain through user agent detection, rather than to optimize the site for mobile searchers. I often get asked, “Why would someone who is on the first page with a working desktop result that redirects to appropriate mobile content in mobile search need to optimize a mobile site for mobile search?” These people typically don’t understand why anyone would need to optimize for mobile search when they do traditional SEO and mobile search engine users are redirected to their desktop site.

It’s a good question, in a sense, because if you have an optimized desktop site that appears in mobile search results, and you’re redirecting mobile users to mobile content, you’re getting some traffic to your mobile site from mobile searchers, even if the site’s not in the search engine’s index. Why would one need to specifically optimize for mobile searchers, when the mobile searchers are finding their content via redirects? I know that a few of the major mobile content publishers simply use redirects rather than optimize for mobile search specifically, and I have to say I think that’s a bit short-sighted. I sat down to think of a good objection to the practice, and came up with sixteen:

1. Mobility is a ranking factor for mobile search, and not optimizing a mobile site could make it more difficult to appear in competitive nonbranded searches where mobility is a factor. Likewise, having an optimized mobile site could make it easier to appear for competitive nonbranded keywords where mobility is a factor.
2. Some users prefer to access desktop sites on their smartphones, and some prefer to access mobile sites. By forcing the mobile site for desktop users, you’re only pleasing the one audience of searchers. Having a link to the mobile site from the desktop site and vice versa, as the W3C recommends, pleases the most users.
3. If online reputation management is a factor for a brand, having a mobile optimized site could give an additional listing (or sometimes two or three) in branded search results, pushing negative content down.
4. Not optimizing a mobile site usually means that a brand is presenting a slimmed down version of their desktop site, which doesn’t take advantage of features that make for an inherently mobile user experience (e.g. GPS or location awareness), which will likely garner fewer natural links than a site that provides a truly mobile user experience, making it more difficult for the site to rank in mobile search.
5. As mobile search evolves and mobile ranking factors become more prevalent, brands that don’t optimize a mobile site today will be busy playing catch up tomorrow, building mobile links and paying attention to mobile ranking factors to their site. Search engines look at age of links and age of site when it comes to ranking because such factors are hard to manipulate. Success in mobile SEO will be difficult if postponed until absolutely necessary.
6. Assuming that mobile SEO is not necessary because a desktop site redirects to a mobile site for mobile browsers assumes that mobile searchers are searching for web sites. Mobile application optimization is a large part of mobile search visibility, and it has very little to do with mobile web search at this point.
7. Voice search, search suggest, location and mobility changes keyword targeting, and a brand with a local presence might be able to target more competitive keywords to an audience that is nearby. This can’t be controlled if a mobile site is not optimized for mobile searchers.
8. Mobile visual search is another emerging mobile search trend that has nothing to do with desktop or mobile web sites. If someone takes a picture of a store, logo or product, will they be able to find the information that they’re looking for? This can be optimized, if not controlled, and brands thinking about mobile SEO will be ready for it before brands who aren’t.
9. For Blackberry users, who represent the majority of smartphone users, as well as feature phone users (who represent the majority of mobile users) Google places a green mobile icon next to mobile search results (e.g. http://www.google.com/m/search?q=espn+mobile), which can increase CTR from users who prefer mobile sites and understand what it represents. For these users, desktop sites might not render at all on their phones, and they may need a mobile optimized site in order to interact with it.
10. Keywords like “coupons” could have a different meaning to people on mobile devices than they do on the desktop, as you can’t redeem printable coupons on your phone. You want to provide a different landing page to these consumers in the search engines so that they’re able to convert.
11. Providing a mobile landing page to a mobile user makes it more likely that they will perform the desired action. By redirecting every mobile searcher to the mobile homepage, you’re performing a similar action to sending all paid searchers to a homepage, which isn’t as relevant to their query and forces them to do extra work before they can convert. Optimizing mobile pages for mobile queries increases the chance of conversion.
12. Because of efforts by Google and other search engines to make mobile search easy and convenient for users, it is growing so quickly that many analysts expect it to outpace desktop search by 2011. In general, brands who make an effort to understand and optimize for the mobile search experience will be better prepared for that day when there are more mobile searchers than desktop searchers.
13. Mobile searchers sometimes never get to search listings because they find the content they’re looking for in the search suggest box. Optimizing for mobile search suggest makes it possible for the user to find what the brand wants them to find without ever accessing a mobile site.
14. New phones come out all the time and adding a new user agent for all of them is additional maintenance. Easier to optimize both a desktop and mobile site and let the engines send the right traffic.
15. If a site contains mobile-specific content that’s not on the desktop site, that content will never be found for mobile-specific searches if mobile site is not indexed.
16. It’s an argument from authority, but I’m going to make it because it’s what many people in this industry listen to. That is, it’s not 2005 anymore where it seems like there are only five of us recommending optimizing for mobile search, while most are advising to wait until next year. At this point many agencies and consultants and even engines are recommending optimizing for mobile searchers. Here’s a partial list:
    • Bruce Clay
    • Razorfish
    • Group M/Outrider
    • Resolution Media
    • iCrossing
    • Google (1, 2, 3, 4)

Have you tried to recommend mobile search optimization only to get asked why bother optimizing a mobile site for mobile search? If you did and you didn’t have a compelling argument in favor of optimizing mobile sites for mobile searchers, now you have 16.

So if not for this question, why aren’t you optimizing for mobile searchers? I’m sure there are other questions that people have and I’d like to answer them all. Ask yours in the comments and if I see it enough I’ll address it in a future post.

In the last few years, I’ve visited a lot of newspaper websites for various projects. These sites are most frequently the online arms of what were once strictly printed local newspapers. When visiting these sites, I’ve been struck by the technical clunkiness of most—they’re typified by poor usability, layouts still closely influenced by traditional print newspaper layouts, dysfunctional on-site search engines, and content management systems hamstrung with badly-formed page templates.

Naturally, these sites are not optimized for search engines nor to make their content readily findable via search. It’s unsurprising that the sites are search-unfriendly. The newspapers probably feel highly conflicted in regards to search—the nostalgic desire for successes experienced in the past have made them grow unhappy with the internet paradigm, and they’ve worked each other up into a frenzy to hold Google responsible for their troubles. It’s hard to expressly invite a perceived enemy into your house on one hand while issuing invective against him on the other.

(I have also encountered newspaper sites which have optimized by some degree. But, these seem fairly few, and even some of them have only taken faltering steps in that direction. The exceptions are some of the biggest players such as the New York Times, Chicago Tribune, Wall Street Journal, Washington Post and other juggernaut newspapers—which are doing professional jobs at optimization.)

I’m very sympathetic to the dilemma newspapers are experiencing. I recall a time not long back when newspapers felt that internet yellow pages companies (“IYPs”) were as much of a threat as Google (see Local Media Face Growing Threat from Local Search Competitors Like Google, Overture and Yellow Pages, New AIM Group Study Reports and Search Engines Make Local Landgrabs, Leave Newspapers Out In the Cold), back when yellow pages companies had a considerable head-start over newspapers in online engagement and ad sales.

Since I used to work at an IYP, I also experienced firsthand what it was like to see a veteran print industry work to evolve to fit in the changing landscape while still being influenced strongly by legacy technologies. Technology wasn’t the only issue: organizational resistance toward seeing where things were headed, or even relatively insightful observations that there might be a risk in not engaging more aggressively also held the IYPs back. Since newspapers perceived the threat beginning such a long time ago, it’s disheartening to see that as a whole they have struggled to develop an effective adaptation for online—particularly the smaller, local market papers.

So, what’s to be done?

While there are a great many areas where online newspaper sites might improve and increase revenue prospects, one of the greatest untapped potentials on newspaper sites in my opinion is the news archive section. Even among poorly optimized newspaper sites, some articles may vanish into a walled-garden archive section at some point, going dark for search engines. Combined with very poor on-site search utilities, it’s as though these articles don’t exist at all for consumers.

I can’t count how many different newspaper sites I’ve visited where I’ve searched for articles which I knew existed, yet the on-site search engines could not locate them. In some cases, the “live” sites had search engines separate from archive search, yet offered no explanation to users as to which should be used and in what cases. Do articles pass into archive after one year? Two? Three? Why can’t the on-site search show them, regardless? In many other cases I’ve found articles by searching in Google, but the article is no longer available when I click through to the newspaper site, and searching within the site fails to reveal it. Did the article “expire” and pass into the archive graveyard or something? No messaging on the resulting error pages reveals this, nor suggests viable means for locating the article.

Newspaper folks: this is your main product! It’s all well and good to try to keep Google from making everything free and putting you out of business, but at this point there’s an even greater danger in locking away your content to the point where online consumers cannot even find—if a searcher doesn’t even know it exists, it’s certain they won’t be engaging with your site to try to obtain it, regardless of whether it’s provided “free” in return for ad impressions, in exchange for “free registration” or provided in return for some subscription fee.

How many articles are locked away in these old archives?!? It surely varies from newspaper to newspaper, but the potential numbers are staggering. While clicks on pay-per-click ads on newspaper sites may add up slowly, there’s no doubt in my mind that if newspapers dramatically expanded the content they have available to search engines, the clicks and associated revenue would increase. These newspapers must not realize the potential they’re sitting upon!

I acknowledge that current news is going to be the more popular content on newspaper sites, but there’s likely at least half as much traffic potential in the legacy content under the theory of the long tail. According to that theory when applied to newspaper website traffic, yes, there’s far more visits per contemporary news story than past ones, but the cumulative traffic from thousands and thousands of past news stories can equal or dwarf the traffic from the more popular stuff.

So, how should news archives be optimized for search?

Here are a few tips to get you started:

Optimize titles & headlines. This is one area where newspapers should utterly dominate! Reporters and editors often write beautiful article headlines which succinctly describe the topic and grab readers’ attention. But, the headline prose is squandered on some newspaper sites which either repeat the newspaper’s name for the TITLE text of all pages, or cram it up with the date, newspaper name and other “branding” messaging before the article title.

This amounts to almost criminal misuse of the title tag. The title is often the link text that’s displayed in search engine results when pages on your site are found to match the search term, and it’s displayed at the top of the browser window when a user visits the page. Also, in HTML there is a particular tag called the “heading” which is intended for just what it sounds like—used as way of identifying the heading and subheadings on a page, and it should be used when displaying article headlines. There are six different heading tags available (each uses different font attributes to add or decrease emphasis), but the main one you need to know is the <h1>, which is perfect for use in displaying an article’s headline on the page. The article headline should also appear at the beginning, not end, of title tags, and be displayed in H1 tags on the page for best usability and search engine friendliness.

Link to all of your content. To this day, search engines still rely heavily on links to pages to discover and index content. For good usability and crawability, I recommend designing a hierarchy of pages on your site so that users may click from the homepage to a page which provides a short list of top level links (such as links by dates or category of type of news story). Those top level links can link down to subcategory pages which link down further to each article ever published by your newspaper. Such a hierarchy of links is mainly for human site users to navigate down into all of your content, but it also helps the search engines understand the site structure, apply relative priority weighting of pages, and also to semantically categorize content.

Create and actively maintain sitemaps. Although the category pages I mentioned above are often loosely referred to as sitemaps, “official” sitemaps files (those created using a formal standard acknowledged by all of the major search engines) are lists of links to your pages that search engines use find all of your content. These should be used in conjunction with the hierarchy of links provided for human users. The sitemaps help insure that the search engines can find all of your pages.

Focus on creating evergreen URLs. Search engines tend to respect pages that have been around for a long while, and frequently changing the physical location of pages confuses both users and search engines alike. So, try to design article URLs which remain stable when an article is pubbed all the way through to when it’s archived for the long term. People also tend to link to articles, which helps search engines to decide how popular a page is. If you change the URL, then the link “votes” for popularity will no longer point to your article.

If you must change URLs, use 301 redirects. If your system is full of legacy processes which require you to change article page URLs once a current news article is moved off into the archive, then at least redirect the original URL to the final location instead of just delivering up an error page. Most users who click through won’t stop to poke around to try to find where something was shifted-to—they’ll just abandon your site to try to find info elsewhere. And, that redirection command should be a 301 “permanent” redirection in order to insure the search engines apply the original URL’s popularity weighting to the new URL.

Offer a “first click free” option. For those newspapers which require registration or subscription prior to showing archive content, read up on Google’s “First Click Free” program. This process allows Google to crawl your site content and index it, and a person who clicks through from Google can view and read the first page for free, but you can then require payment or registration for subsequent pages.

Create a subscription designation with Google. If you set it up with them, Google will allow you to have content crawled, but when users click through they must pay or register to see any of the article. This is less-preferred by Google since it’s a less satisfactory user-experience. If going this route, I’d recommend displaying a good-sized chunk of the article to users that click through, as a preview. In that way, they’ll feel a little less disappointed, and may be drawn in further to pay a subscription to see more.

Improve on-site search Just as a usability matter, consider using Google’s site search if your internal site search doesn’t work well. Once the pages have been optimized as I’ve outlined above, you could implement Google site search and perhaps improve your site’s overall usability.

There are certainly many other areas for optimization for newspapers, and this is not an exhaustive list of SEO improvements which could be done for articles. Simply exposing hidden and non-indexed archives would be a good start.

Google is trying to find additional ways in which to help the newspaper industry, such as its plan to roll out a micropayment system within a year. Micropayment is FAR better, in my opinion, than attempting to require someone to purchase a month-long or yearly subscription when they might want to access only a single article. I’d also suggest improving classified sections or partnering with many news sites for a multi-site subscription.

The local newspapers have lost a lot of marketshare to online news sites and aggregators, but evolving to improve their popularity and traffic in the internet economy could help them to take back marketshare and increase revenue. SEO helps with promotion and expansion of audience. If you’re a local newspaper in need of increased business, seriously consider beefing up your search engine optimization game.

Here are some of the types of changes that can have an impact on traffic or rankings.

Domain change. Any change in the domain, such as a move from http://www.old-domain.com to http://www.new-domain.com. The most common reason for doing this is a branding change of some sort. An existing business may be changing its branding, or one business entity may have been acquired by another one and the two sites are being merged.

Structural changes or URL changes. These are changes where the content that lives on a given URL on old-domain.com (such as about-us.html) gets moved to a different URL (such as about-us.php). URL changes can be “wholesale” (change nearly all or all of them), “heavy” (change a lot of them), “moderate” (change some of them), “light” (change only a few), or not done as all if you simply copy the exact site structure from one domain to another.

Structural changes often happen as a result of a change in the technology used to implement a site. For example, a business may have been using Cold Fusion as a content management system, and then switches to using ASP. The other major reason for structural changes is when wholesale content changes are made.

Content changes. Changes to the content on pages can happen without changing the URL structure of the site, by simply rewriting content on the pages, or something that causes structural changes to the site. As with URL changes, these can also be heavy, moderate, light, or not done at all.

Content changes may be made for many reasons. Perhaps the target audience has changed. Perhaps the basic positioning of the organization has changed. Another possible reason is to revamp the content as part of a wholesale expansion of the site.

Each of these things can happen independently. You can make content changes without changing the domain or the URLs. You can change the URLs without changing the domain or the content.

What are the true consequences?

You are going to lose traffic. That is a fact. Even if you only perform a domain change and preserve the exact same site structure and content, you will lose some traffic. In this simplest of scenarios you can minimize the amount of traffic loss by using 301 redirects from each URL on the old domain to point to the same URLs on the new domain, alerting the search engines that the new URLs are the important ones.

In principle, this simple domain change scenario sounds like one where there should be very little lost traffic. One factor to consider though is that of “trust.” Any time there is a domain change it may be reflective of an ownership change, even if the WhoIs info is not updated. For the search engines this raises the possibility that the new owner isn’t as trustworthy as the original owner.

Another factor concerns the 301 redirects themselves. In tests we have done at Stone Temple Consulting, we have seen evidence that they pass through the majority, but not all, of the link juice to the destination page. Sometimes there is a delay between the implementation of the redirect and when the search engines pass through the link juice, which can result in a significant drop in search engine traffic. In the medium to long term a simple domain change is usually not that costly (though there are exceptions). You may lose 20% to 40% of your traffic in the short term, and 10% to 20% in the medium to long term.

As you might imagine, the more complex the changes, the greater the potential negative consequences. For example, combining a domain change with URL changes will definitely be more costly. You have given the search engine more reasons to trust the site less, and your 301 redirect map just got more complicated. Assuming you completely restructure the site so all the URLs change, you can expect to see traffic loss of about 30% to 50% traffic loss in the near term, with gradual improvement on that over the longer term.

In our final scenario, if you change your domain, URL structure, and your content, you are asking for trouble. The big reason for the cost here is that the new content you create is not the content that people saw when they linked to your site in the past, even if it basically about the same subject matter. This probably results in the search engine significantly discounting the value of those links. Traffic loss in this scenario is likely to be 50% or more in both the short and long term.

To summarize:

Disclaimer! These numbers aren’t exact, and your mileage will vary. The actual impact of changes to your site will depend on many factors that are not possible to cover here. In addition, the chart assumes that you don’t do a lot of incremental link building to bolster rankings. However, savvy site owners rarely stand completely still.

How to mitigate the damage

Once you have made your changes, and assuming you have followed Google’s recommended best practices for doing so, the main damage control you can do is get new links to the site. In particular, if you can continue to get links at a pace similar to, or better than, what was happening before the move, this is a strong positive signal to the search engines that all is well.

Also make sure that you ask people who have linked to you in the past to update their links to go direct to the new site, bypass those pages where you’ve put 301 redirects in place. If a significant percentage of your past linkers do this it is also a very strong signal to the search engines that your site is still trustworthy in its new location.

The best remedy? Stop and think about the consequences of a move before committing. There will be a cost, and your business plan probably does not call for a dip in traffic to, and orders from, the web site.

The bottom line: Don’t make major changes to your site unless you really have to, and are willing to pay the price.

Fetch as Googlebot

Of most interest to webmasters, SEOs, and web developers is likely the Fetch as Googlebot feature. You can specify any URL on your site and see the HTTP response (header and contents) that the server returns. Simply  indicate the URL and click the Fetch button. It may take a few moments for Googlebot to access the page and return the results, since it fetches the page in real time. (Refresh the page to see the progress.)

Click the Success link once it’s been processed to see the results.

How is this different from simply looking at the source code of the page?

• You see the HTTP header information at the top. This information is generally easily available through tools such as Live HTTP Headers, but isn’t contained in the source code itself (since that information is coming from the server, not the page).
• You can see if the server is returning any of the page information differently than the page has been coded.
• You can see if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (typically known as “cloaking“), you’ll see  what’s being conditionally served to Google.
• You can use the tool to test changes (particularly things like redirects) in real-time.

Note that this tool won’t necessarily show you the content that Google is able to extract from the page. If the page contains JavaScript, for instance, you’ll see the raw JavaScript code contained on the page, not the rendered view visible in the browser. Which, unfortunately means you can’t use this tool to determine if Google is able to access content contained in rich markup.

What’s this about cloaking?

This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re coming into a project late and aren’t sure what’s been previously done. It can also help uncover if your site has been hacked. Back in 2006, Googler Matt Cutts and I did a show on Webmaster Radio during which we talked about how in some cases, a hacker might add links to a site and then cloak those pages so that the site owner never sees them. Only Google does. At the time, Matt suggested using Google Translate (and choosing English to English) to see what Googlebot was being served, but this tool can now more easily serve that purpose. Matt confirmed this to me this morning: “The biggest use case is just debugging site issues. Of those, the biggest case will be hacked sites. Some attacks will hide content until search engines fetch the page (and some attackers add a noarchive tag so that the search result doesn’t have a “Cached” link), so a site could look clean to the website owner. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”

How do I test redirects?

If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. For instance, when I fetch www.searchengineland.com, I see that the redirect is correctly implemented as a 301 and points to searchengineland.com:

You can also use the tool to troubleshoot URLs listed in the Crawl Errors > Not Followed report. You can also test these URLs using something like Live HTTP Headers or by trying to access the URLs in a browser, but if neither of those methods uncover the problem, this tool can help determine that the issue is specific to Googlebot. You can also use this tool to verify that fixes you’ve made to redirect errors uncovered by the Not Followed report have really solved the problem.

(Note that the tool currently has a limit of 100kb per page. However, this is for the tool only and doesn’t apply to Googlebot’s normal crawl of the site. Google is monitoring feedback to see if many site owners find this size to be limiting.)

Malware details

The Google Online Security Blog has more information on the malware details tool. Previously, webmaster tools reported when the site was flagged has having malware and listed sample URLs. This new tool will also show samples of the malicious content, and in some cases, the underlying cause. This should help those site owners whose sites have been hacked to include malware find the problem and fix it. If your site does contain malware and you’ve fixed it, you can request a review to have the malware alert removed in search results.