It’s no secret that newspapers have been struggling with the disruptive innovations introduced by the internet, and this has resulted in some level of accusation towards Google and a circling-of-the-wagons mentality by the industry. But, what if they were to go in the opposite direction, with fuller engagement? Here’s one tactic for how to go about it via SEO.

In the last few years, I’ve visited a lot of newspaper websites for various projects. These sites are most frequently the online arms of what were once strictly printed local newspapers. When visiting these sites, I’ve been struck by the technical clunkiness of most—they’re typified by poor usability, layouts still closely influenced by traditional print newspaper layouts, dysfunctional on-site search engines, and content management systems hamstrung with badly-formed page templates.

Naturally, these sites are not optimized for search engines nor to make their content readily findable via search. It’s unsurprising that the sites are search-unfriendly. The newspapers probably feel highly conflicted in regards to search—the nostalgic desire for successes experienced in the past have made them grow unhappy with the internet paradigm, and they’ve worked each other up into a frenzy to hold Google responsible for their troubles. It’s hard to expressly invite a perceived enemy into your house on one hand while issuing invective against him on the other.

(I have also encountered newspaper sites which have optimized by some degree. But, these seem fairly few, and even some of them have only taken faltering steps in that direction. The exceptions are some of the biggest players such as the New York Times, Chicago Tribune, Wall Street Journal, Washington Post and other juggernaut newspapers—which are doing professional jobs at optimization.)

I’m very sympathetic to the dilemma newspapers are experiencing. I recall a time not long back when newspapers felt that internet yellow pages companies (”IYPs”) were as much of a threat as Google (see Local Media Face Growing Threat from Local Search Competitors Like Google, Overture and Yellow Pages, New AIM Group Study Reports and Search Engines Make Local Landgrabs, Leave Newspapers Out In the Cold), back when yellow pages companies had a considerable head-start over newspapers in online engagement and ad sales.

Since I used to work at an IYP, I also experienced firsthand what it was like to see a veteran print industry work to evolve to fit in the changing landscape while still being influenced strongly by legacy technologies. Technology wasn’t the only issue: organizational resistance toward seeing where things were headed, or even relatively insightful observations that there might be a risk in not engaging more aggressively also held the IYPs back. Since newspapers perceived the threat beginning such a long time ago, it’s disheartening to see that as a whole they have struggled to develop an effective adaptation for online—particularly the smaller, local market papers.

So, what’s to be done?

While there are a great many areas where online newspaper sites might improve and increase revenue prospects, one of the greatest untapped potentials on newspaper sites in my opinion is the news archive section. Even among poorly optimized newspaper sites, some articles may vanish into a walled-garden archive section at some point, going dark for search engines. Combined with very poor on-site search utilities, it’s as though these articles don’t exist at all for consumers.

I can’t count how many different newspaper sites I’ve visited where I’ve searched for articles which I knew existed, yet the on-site search engines could not locate them. In some cases, the “live” sites had search engines separate from archive search, yet offered no explanation to users as to which should be used and in what cases. Do articles pass into archive after one year? Two? Three? Why can’t the on-site search show them, regardless? In many other cases I’ve found articles by searching in Google, but the article is no longer available when I click through to the newspaper site, and searching within the site fails to reveal it. Did the article “expire” and pass into the archive graveyard or something? No messaging on the resulting error pages reveals this, nor suggests viable means for locating the article.

Newspaper folks: this is your main product! It’s all well and good to try to keep Google from making everything free and putting you out of business, but at this point there’s an even greater danger in locking away your content to the point where online consumers cannot even find—if a searcher doesn’t even know it exists, it’s certain they won’t be engaging with your site to try to obtain it, regardless of whether it’s provided “free” in return for ad impressions, in exchange for “free registration” or provided in return for some subscription fee.

How many articles are locked away in these old archives?!? It surely varies from newspaper to newspaper, but the potential numbers are staggering. While clicks on pay-per-click ads on newspaper sites may add up slowly, there’s no doubt in my mind that if newspapers dramatically expanded the content they have available to search engines, the clicks and associated revenue would increase. These newspapers must not realize the potential they’re sitting upon!

I acknowledge that current news is going to be the more popular content on newspaper sites, but there’s likely at least half as much traffic potential in the legacy content under the theory of the long tail. According to that theory when applied to newspaper website traffic, yes, there’s far more visits per contemporary news story than past ones, but the cumulative traffic from thousands and thousands of past news stories can equal or dwarf the traffic from the more popular stuff.

So, how should news archives be optimized for search?

Here are a few tips to get you started:

Optimize titles & headlines. This is one area where newspapers should utterly dominate! Reporters and editors often write beautiful article headlines which succinctly describe the topic and grab readers’ attention. But, the headline prose is squandered on some newspaper sites which either repeat the newspaper’s name for the TITLE text of all pages, or cram it up with the date, newspaper name and other “branding” messaging before the article title.

This amounts to almost criminal misuse of the title tag. The title is often the link text that’s displayed in search engine results when pages on your site are found to match the search term, and it’s displayed at the top of the browser window when a user visits the page. Also, in HTML there is a particular tag called the “heading” which is intended for just what it sounds like—used as way of identifying the heading and subheadings on a page, and it should be used when displaying article headlines. There are six different heading tags available (each uses different font attributes to add or decrease emphasis), but the main one you need to know is the <h1>, which is perfect for use in displaying an article’s headline on the page. The article headline should also appear at the beginning, not end, of title tags, and be displayed in H1 tags on the page for best usability and search engine friendliness.

Link to all of your content. To this day, search engines still rely heavily on links to pages to discover and index content. For good usability and crawability, I recommend designing a hierarchy of pages on your site so that users may click from the homepage to a page which provides a short list of top level links (such as links by dates or category of type of news story). Those top level links can link down to subcategory pages which link down further to each article ever published by your newspaper. Such a hierarchy of links is mainly for human site users to navigate down into all of your content, but it also helps the search engines understand the site structure, apply relative priority weighting of pages, and also to semantically categorize content.

Create and actively maintain sitemaps. Although the category pages I mentioned above are often loosely referred to as sitemaps, “official” sitemaps files (those created using a formal standard acknowledged by all of the major search engines) are lists of links to your pages that search engines use find all of your content. These should be used in conjunction with the hierarchy of links provided for human users. The sitemaps help insure that the search engines can find all of your pages.

Focus on creating evergreen URLs. Search engines tend to respect pages that have been around for a long while, and frequently changing the physical location of pages confuses both users and search engines alike. So, try to design article URLs which remain stable when an article is pubbed all the way through to when it’s archived for the long term. People also tend to link to articles, which helps search engines to decide how popular a page is. If you change the URL, then the link “votes” for popularity will no longer point to your article.

If you must change URLs, use 301 redirects. If your system is full of legacy processes which require you to change article page URLs once a current news article is moved off into the archive, then at least redirect the original URL to the final location instead of just delivering up an error page. Most users who click through won’t stop to poke around to try to find where something was shifted-to—they’ll just abandon your site to try to find info elsewhere. And, that redirection command should be a 301 “permanent” redirection in order to insure the search engines apply the original URL’s popularity weighting to the new URL.

Offer a “first click free” option. For those newspapers which require registration or subscription prior to showing archive content, read up on Google’s “First Click Free” program. This process allows Google to crawl your site content and index it, and a person who clicks through from Google can view and read the first page for free, but you can then require payment or registration for subsequent pages.

Create a subscription designation with Google. If you set it up with them, Google will allow you to have content crawled, but when users click through they must pay or register to see any of the article. This is less-preferred by Google since it’s a less satisfactory user-experience. If going this route, I’d recommend displaying a good-sized chunk of the article to users that click through, as a preview. In that way, they’ll feel a little less disappointed, and may be drawn in further to pay a subscription to see more.

Improve on-site search Just as a usability matter, consider using Google’s site search if your internal site search doesn’t work well. Once the pages have been optimized as I’ve outlined above, you could implement Google site search and perhaps improve your site’s overall usability.

There are certainly many other areas for optimization for newspapers, and this is not an exhaustive list of SEO improvements which could be done for articles. Simply exposing hidden and non-indexed archives would be a good start.

Google is trying to find additional ways in which to help the newspaper industry, such as its plan to roll out a micropayment system within a year. Micropayment is FAR better, in my opinion, than attempting to require someone to purchase a month-long or yearly subscription when they might want to access only a single article. I’d also suggest improving classified sections or partnering with many news sites for a multi-site subscription.

The local newspapers have lost a lot of marketshare to online news sites and aggregators, but evolving to improve their popularity and traffic in the internet economy could help them to take back marketshare and increase revenue. SEO helps with promotion and expansion of audience. If you’re a local newspaper in need of increased business, seriously consider beefing up your search engine optimization game.

Many site owners have wanted the recently introduced canonical tag to work across domains. Now their wishes will come true. Google announced cross-domain support will come by the end of the year.

The news came out during the Duplicate Content: The Search Engines Edition session at SMX East conference today. Google made the announcement in response to site owners again voicing that they wanted such a solution.

The existing canonical tag is only supported by Google, at the moment. Yahoo and Bing both said they’re studying support but think its likely they’ll add support by the end of the year. However, they’ll only support canonicalization across the same domain.

In a way, they’ll support Canonical Tag 1.0 by the end of the year, while Google will be supporting Canonical Tag 2.0, with cross-domain support.

We will have more information when more details come to us.

Each day millions of online shoppers turn to comparison shopping engines (CSEs) to compare prices, read reviews, research products and make buying decisions. While comparison shopping engines don’t sell or ship the products featured on their sites, they advertise products from merchants to a diverse online audience. Merchants are using CSEs in a down economy to capture shoppers looking for deals. To take advantage of this growing advertising space and get more exposure than your competitors, you need a product data feed—and even more importantly, you must optimize the feed to gain maximum exposure in CSE search results.

The number of online shoppers is increasing, and they’re internet-savvy. They will scan a product page for what interests them—the best price, a brand they trust, free shipping or seasonal discounts. It’s no surprise that clickthrough rates (CTR) and conversions increase when:

• Titles are relevant: Is this what I am searching for?
• Descriptions are engaging: Is this the right size, color, style?
• Promotions are enticing: Should I buy this now?

Perfect your listings! Making the leap from interest to conversion

Here are some specific tips for optimizing feeds to maximize your exposure in shopping engine results:

Craft targeted, keyword focused titles. On search engines, the online shopper controls what they see. They will search for brand, model, product type and product attributes. Shopping engines bold the keywords in your product title that match the shopper’s query. Bolded keywords in the product title increase the click attractiveness of your listing and also qualify your lead.

Map your products to the taxonomy of each CSE. Loose classifications lead to fewer and poor quality clicks, but what will hurt your campaign most is the missed opportunity of qualified shopping traffic that could easily have been yours if your products were correctly categorized. If you sell iPods and submit them as electronics in your feed, your competitors are likely to outperform you, taking sales that might have been yours.

Products that are not mapped to the right taxonomy, are not granular in their categorization, or are left uncategorized are at a disadvantage because more closely mapped products will climb in the rankings and get a disproportionate number of impressions.

Include detailed product specifications, attributes and features for each product. Detailed product specifications will surface your listing more frequently for targeted keywords with higher shopping intent, leading to more conversions. Here is a great example: “This energy efficient coffee grinder makes grinding coffee beans an easy task. Features stainless steel blades, easy access cord storage and 2-ounce capacity for 12 cups of coffee. Great for making flavored coffees.” If a shopper searches for a 12 cup coffee maker, this merchant will surface.

Cross-sell related products.: Include product uses such as “can be used with,” “best suited for,” and so on so your product may be displayed as a related product or “shoppers also viewed,” increasing visibility and appeal to shoppers.

Add visual prominence with images and logos. Large images get more attention and are the first to be picked for featured pages and shopping blog entries. Shoppers love close-up images and pictures that show multiple angles of the product. Even if CSEs display a thumbnail image on the search results, if you provide a full size image of the product most will offer shoppers the ability to click through to the full size image of your product. In the blog post below, the blogger has chosen to feature a picture with multiple views of trail running shoes to provide a more detailed visual for the write-up.

CSEs may also allow you to add your logo for an additional charge, a feature that gets attention and lends credibility to your product listing.

Tune your marketing message. Savvy shoppers are always looking for deals, discounts, specials, free shipping or other savings online. If you offer specials, provide these promotions in the feed to CSEs. Products that display promotions will not only attract more attention on product search pages, but will get highlighted on featured deals and promotions pages. Many CSEs feature specials on their blog, tweet about them or highlight the promotion on a “daily deals” page. Some CSEs offer a promotional text feature, allowing you to advertise these deals with your products listings. In the PriceGrabber deal of the day below, the “Planet Earth – Complete Collection” Blu-ray DVD is featured as the deal of the day.

Don’t spend—invest. Spending gets cut in a tight economy, but investments continue. Track your ROI. As long as you’re seeing return, it is an investment. Work with your CSE to closely track conversions on the traffic you receive. If possible, consider installing any available ROI tracker the CSE provides to better analyze performance. When the economy recovers, your margins will multiply because you will know exactly where to invest for greater returns.

Promote hot products and seasonal items in your feed. CSEs want to help their retailers promote their products, and this data is important! Surface your seasonal products in the feed and call them out. Most CSE’s actively scour their products for seasonal items and promote them on special seasonal features, such as guides, newsletters, press releases, shopping blogs, tweets and more.

Strive to get in the top three listings. Whether on a search engine like Google or a CSE, the top three results get the lion’s share of qualified leads. Analytics show that even on shopping sites, conversions for the top three listings are disproportionately high. Quality score matters on shopping engines too. In addition to bidding, relevance and popularity impact the position of your listing. A placement above the fold gets a disproportionate number of eyeballs from shoppers, so it’s important to optimize your feed.

Make it stick. Increase your branding and visibility to consumers through other advertising channels like banner advertising or page sponsorship. Just as paid ads coupled with organic search results in Google boost traffic to the site, banner ads on comparison shopping sites improve branding and traffic to your site.

Merchants who pay attention to optimizing their feeds and leverage the advanced features and new opportunities offered by CSEs will position themselves for increased traffic and conversions.

The Google Webmaster Central blog notified us that Googlers have presented a new study on Sitemaps at the WWW’09 conference in Madrid. The study is absolutely interesting and I recommend printing out the ten page PDF document and reading it. For those of you who don’t have time for that, I hope to highlight the most interesting findings from the study below.

The purpose of the study was to measure the past few years of Sitemaps usage at Google to determine how Sitemap files improve coverage and freshness of the Google web index. By coverage, I mean how Google crawls the web deeper and finds more content that it might not have found. Bt freshness, I mean how Google crawls new or updated content faster, when compared to the normal crawl.

Interesting facts from the study:

• ~35 million Sitemaps were published, as of October 2008.
• The 35 million Sitemaps include “several billion” URLs.
• Most popular Sitemap formats include XML (77%), Unknown (17.5%), URL list (3.5%), Atom (1.6%) and RSS (0.11%).
• 58% of URLs in Sitemaps contain the lastmodification date.
• 7% of URLs contain the change frequency field.
• 61% of URLs contain the priority field.

The paper discusses the process used by Google for Sitemaps. Here is a flow diagram that explains it quickly.

Coverage: The dataset used to measure the “coverage” of Sitemaps was approximately 3 million URLs, 1.7 millions URLs specifically from Sitemaps and the remainder from the normal discovery process. Duplicate URLs were close to one million during the discovery crawl process, as opposed to only a 100 duplicate URLs in the Sitemaps files. In short, the study found that discovery was 63% “efficient” and Sitemaps was 99% efficient in crawling the domain at the cost of mission a small fraction of content.

• The percent of duplicates inside Sitemaps is mostly similar to the overall percent of duplicates.
• 46% of the domains have above 50% UniqueCoverage and above 12% have above 90% UniqueCoverage.
• For most domains, Sitemaps achieves a higher percent of URLs in the index with less unique pages.

Freshness: How fresh can Google get with Sitemaps?

• 78% of URLs were seen by Sitemaps first, compared to 22% that were seen through discovery first.
• 14.2% of URLs are submitted through ping
• The probability of seeing a URL through Sitemaps before seeing it through discovery is independent of whether the Sitemaps was submitted using pings or using robots.txt

The paper then goes on to talk about coming up with ways to determine the crawl order, either via Sitemaps or Discovery. Concepts such as SitemapScore and DiscoveryScore are brought up and possible methods.

The study seems like a great read for most SEOs interested in understanding how Google Sitemaps work and how it can benefit your sites.

A month ago, Google, Yahoo and Microsoft announced they will be supporting a new canonical tag that allows you to tell search engines that page X is a duplicate page to page Z. In a way, it is a 301 redirect, without the physical redirect.

The tag is incredibly powerful, as are 301 redirects and using this tag should be done with caution and slowly. Matt Cutts posted a new video explaining how one should go about using this tag, being that it is so new. Here is the video:

The Google Webmaster Central blog announced that they have just released a new Sitemap generator that works on both Linux/Apache and Microsoft IIS Windows-based servers. The Sitemap generator is different and possibly smarter than other Sitemap tools because it uses web traffic, log files and other methods to find new or modified URLs. You can download the new tool over here.

Why go through the process of setting up a Sitemap file? Well, Google themselves said on the Sitemap Generator page, “you are more likely to get better freshness and coverage in search engines” if you submit a Sitemap to Google. Plus, there are some reports to validate that claim.

An SEOMoz post has documented statistics on how submitting a Sitemap to Google and Yahoo impacts the rate of crawling that web site. The results show that submitting a Sitemap to Google and Yahoo decreased the time it took Google and Yahoo to crawl the page.

It is important to note that the test was done “on 12 different posts, 6 with Sitemaps being submitted, and 6 with the Sitemaps not being submitted.” The results were that when a “Sitemap was submitted, the average time it took for the bot to visit the new post was 14 minutes for Google and 245 minutes for Yahoo.” In contrast, when “no Sitemap was submitted and the bot had to crawl to the post, it took 1375 minutes for Google and 1773 for Yahoo.” That is a huge difference in crawl time. It is true that many sites see their content indexed within minutes, with or without a sitemap file. But in some cases, it takes longer.

In the old days, I was not a fan of submitting sitemap files for sites that were very search engine friendly. But sometime last year, I became a fan of using sitemap files for search engines. Mainly because it gives me an extra metric in the webmaster tools area to glean information. But in addition, I have been hearing positive results from webmasters after using it over the course of 2008. Finally, I have been noticing that Google is pouring more and more time into these tools and using the data to improve the search results – so why not be there in most cases?

The major search engines have announced an update to the sitemaps.org protocol which enables site owners to store their XML Sitemap files in any location — even on a different domain than the one referenced in the Sitemap. This will be a welcome change for those who manage multiple domains and would like to keep all Sitemap files in one place, as well as for those who would like to store their Sitemap in a location other than the root.

The only caveat? You have to be able to edit the robots.txt file of the domain the Sitemap file references.

The search engines made the announcement today on the Search Engineers Q&A panel at SMX West. Below, more about how this works and how to implement it on your site.

Historically, your Sitemap file had to be stored in the same location as the URLs listed in that Sitemap. For instance:

http://www.example.com/sitemap.xml could include: www.example.com www.example.com/folder1/page1.html www.example.com/folder2.page1.html

But it could not include: subdomain1.example.com www.example2.com

Similarly, http://www.example.com/folder1/sitemap.xml could include: www.example.com/folder1/page1.html www.example.com/folder1/images/page2.html

And could not include: www.example.com www.example.com/folder2/page1.html

The protocol required this to ensure that whoever could modify the Sitemap file also had ownership rights of the URLs listed in the Sitemap. However, this setup was problematic for some sites, including those with multiple domains and subdomains who would have found it difficult or technically infeasible to write Sitemap files to multiple locations. Some sites also found it technically different to write Sitemap files to the root of the domain.

With Google, site owners can overcome this issue by verifying ownership of all domains in Webmaster Central, then submitting Sitemaps for any of those domains from any location. However, as this solution only works for Google, it’s not useful for those who want one process for generating Sitemaps for submission to all engines. (Which is, after all, the spirit of sitemaps.org.)

With the latest sitemaps.org update:

• Each Sitemap file can reference only one domain.
• The Sitemap can be placed anywhere (on a different domain or in a subfolder) as long as its location is referenced in the domain’s robots.txt file.

Referencing The Sitemap File In robots.txt Simply point at the Sitemap location in your robots.txt file. For instance, if you have a Sitemap file for www.example and you want to store it at www.example1.com/sitemap.xml, add to the following line to www.example.com/robots.txt:

Sitemap: http://www.example1.com/sitemap.xml

This line lets the search engines know that you have ownership of www.example.com (since you can modify the robots.txt file for it) and that the Sitemap for that domain is located at http://www.example1.com/sitemap.xml.

If you want to store Sitemaps for multiple domains in one location, simply create a Sitemap for each domain, then modify the robots.txt for each of them to point at the Sitemap file location, as shown below:

The same technique works if you want to store your Sitemap in a location other than the root. If you want to store the Sitemap at http://www.example.com/sitemap/sitemap.xml, simply modify the robots.txt file for www.example.com with the following:

Sitemap: http://www.example.com/sitemap/sitemap.xml

For more information, see the sitemaps.org documentation or each search engine’s blog post:

Microsoft Live Search Google Webmaster Central Yahoo! Search Blog

The Google Webmaster Central Blog has posted a Sitemaps FAQs blog post. In this blog post, Google responds to some of the most asked questions on the Sitemaps protocol and how Google supports it.

Some of the questions include:

• Does a Sitemap submission guarantee page inclusion?
• Can submitting a Sitemap hurt my site?
• Will my rankings improve if I submit a Sitemap?

Check out the full FAQs over here.

Mickey Kataria has posted on the Google Webmaster Central blog that if you can verify ownership of multiple sites, you can consolidate the Sitemap files for those sites and either combine the URLs from those sites into one Sitemap or create separate files for each site, but store the files in one location. Previously, any Sitemap you submitted could contain only URLs from the site on which the Sitemap was hosted.

One instance in which this new functionality may be useful is sites with subdomains. It may be easier to write one file that contains URLs from all subdomains on a site. In this case, you would need to verify ownership of each subdomain separately, but then could create one Sitemap that contains URLs from each subdomain and submit that single Sitemap. The Google Webmaster Help Center has more details.

Sitemaps are a joint standard with all major search engines and Mickey notes in the blog post that while they are talking to the other engines about standardizing this functionality, it currently works for Google only.