Near the end of the talk, someone asked if how much money Google will make is factored into decisions about changes to Google’s (unpaid search) algorithms. Singhal was adamant: “no revenue measurement is included in our evaluation of a rankings change.” Listening to him explain how excites he gets about search improvements and how changes are evaluated, you realize there’s no spin here. He’s absolutely telling the truth. And he would know. Chris Sherman asked if anyone at Google really understands how the whole thing works and he replied that while no one knows how everything works (all of unpaid search, AdWords, Android, etc.), he has a pretty good idea of how all of unpaid search works. Not many can make that claim.

Core to Singhal’s talk was a focus on what Google does look at when improving unpaid search algorithms. The key is always relevance.

Singhal talked about the evolution of Google’s unpaid search algorithms. In 2003, they worked on stemming and synonyms. This meant that those searching for [watch buffy the vampire slayer] [watching buffy the vampire slayer] and [view buffy the vampire slayer] would likely all see the same results. In 2007, came universal search, which was a big step forward in understanding searcher intent. (Searchers typing in [i have a dream] not only are looking for Martin Luther King Jr.’s speech,  but would like to see a video of it.)

Understanding Intent

Ten years ago, search results were keyword-based, but Google is now moving towards understanding the intent behind the words. Singhal talked about Google’s acquisition of the company FreeBase, which has done substantial work on understanding phrases as entities rather than strings. “Mount Everest” isn’t just two words, it’s also a mountain, with a height, in a location, and so on. (Shortly after the talk, Google launched their Knowledge Graph, which is the next step in this understanding.) Combine intent with speech recognition and mobile devices and you almost end up with what Singhal first glimpsed years ago on Star Trek. We do indeed, live in the future (almost).

Personalization

In 2012 took a big step (whether or not that step was forward is up for debate) towards greater personalization with Search Plus Your World, which began incorporating Google+ into search results for those logged in. Singhal explained that Google+ integration was not the point, it was just a proof of concept. The point was a foundation for a wider world of (more secure) searching over everything: both what’s public in the world and what’s private to each searcher. Perhaps one day Google will in fact be able to find your car keys.

Singhal said that searcher click behavior shows that searchers are happy with this integration. But he acknowledged there’s work to be done. When asked when it would launch in Europe, he said that based on feedback, it’s undergoing improvements first.

Relevance and Data: How Changes Are Evaluated

Search Plus Your World is built and evaluated the way all ranking algorithm changes are: build, evaluate, launch, learn, improve, repeat. Relevance is key to every measurement. Singhal stepped through the process:

1. An engineer at Google has an idea of a signal (one of over 200) that might be introduced or tweaked to improve overall relevance.
2. That algorithm change is run on a test set of data and if all looks good, human raters look at before and after results for a wide set of queries (a kind of manual A/B test). The human raters don’t know which is the before and which is the after. The raters report what percentage of queries got better (more relevant) and what percentage got worse (less relevant).
3. This process gets looped several times as the algorithm is tweaked to better serve results for the queries in the “worse” set.
4. Once the overall manual ratings show that the algorithm tweak makes results better overall, it’s all tested again. This time, a data center (one of many that contains Google’s index and serves results to searchers) is loaded with the new algorithm and a very small slice of searchers (typically 1%) see the modified result set. Are those searchers happier than the ones seeing the version of results without the tweak? Singhal says they compare where searchers click. Clicks on higher ranked pages mean results at the top are likely more relevant, and searchers are happier. (He didn’t say so, but they may look at other data, such as click and back behavior.)
5. An independent analyst compiles the results and provides a statistical analysis, which is presented at a search quality meeting, where engineers look at the data and debate the change. If they decide this tweak improves the quality of search results overall (and is good for the web and doesn’t overly tax internal systems), the change goes out.

This process is happening all of the time with lots of different proposed tweaks and tests. 525 algorithm changes were launched in 2011. That may seem like a lot, but earlier this year Singhal noted that many more changes were tested.

“Concurrently we have approximately 100 ideas floating around that people are testing – we test thousands in a year. Last year we ran around 20,000 experiments. Clearly they don’t all make it out there but we run the process very scientifically.”

Aggregated data from millions of searchers typing millions of queries provides clear patterns. Singhal said that not only do those who get better results more quickly click higher in the search results, but they also search more. (We’ve heard this before from Google. Marissa Mayer, for instance has noted that a half a second delay in rendering search results resulted in 20% fewer searches).

Singhal noted that the kind of personalization platform envisioned with Search Plus Your World is harder to test. Human evaluation looks at relevance, but personal relevance is unique for each searcher. All Google really has to go on is click behavior. Singhal talked with Danny Sullivan about this dilemma a few weeks after Search Plus Your World launched:

“Every time a real user is getting those results, they really are delighted. Given how personal this product is, you can only judge it based on personal experiences or by aggregate numbers you can observe through click-through.”

All of this gets complicated by varied screen size. The user interface becomes more important as increased use of mobile devices and tablets shrink screen real estate.

If these changes are all about increased relevance, why is only Google+ represented in Search Plus Your World? Why not Facebook and Twitter? Singhal explained that most personally useful Facebook data is locked behind a login, and Twitter produces content at a rate that is too massive for Google to crawl quickly and comprehensively. Or, they could, but it would probably take down the Twitter servers. Twitter has also had some technical issues that have made crawling difficult, although are being fixed.

What About Panda and Penguin?

Singhal said that Google’s algorithms aren’t perfect (hence the 20,000 experiments a year). He looks at bad queries every day (and encouraged the audience to let him know about them! So, add them to comments to this post and we’ll forward them along). But when asked specifically about Panda and Penguin, two of the latest high profile algorithm changes, he said that data has shown they significantly improved the number of high quality sites being returned in results. They are not only refining what signals they use in ranking, but are improving how they gather and tune the signals themselves (so signal quality is higher). They are  constantly looking for aberrations in signals.

At the end of the day, he said, site owners need to take a hard look at what value their sites are providing. What is the additional value the visitor gets from that site beyond just a skeleton answer? Ultimately, it’s those sites that provide that something extra that Google wants to showcase on the first page of search results.

What’s In the Top Search Queries Report

First, a refresher on what this data is all about. The top search queries report (available in Google webmaster tools for sites you’ve verified ownership of by selecting Your site on the web > Search queries) lists the top queries that brought traffic to your site from Google organic search (from all countries and properties).

Summary Data

For the selected data range, the report shows the total number of queries that brought traffic to the site, the total impressions and clicks the site received, and the number of impressions and clicks for the reported top queries.

Query-Specific Data

For each query, the report notes:

• Number of impressions - how many searchers saw the site in search results for that query
• Number of clicks - how many searchers clicked on the search result for that query
• Click through rate - The percentage of the time searchers who saw the site in search results for that query clicked on it
• Average position - the average position the highest ranked URL for that site appeared in search results for the query across all searchers
• In addition, you can find out the change for each of these data points from the previous period. However, the change percentages aren’t available for time periods longer than 30 days. The change details used to be visible by default, but they’re now off by default. You’ll need to click the With Change button to see them in the report (although they’ll be included automatically with the CSV download). If you have the change percentage displayed, you’ll need to turn that off in order to expand the date range beyond 30 days.

You can click into any query to get more specific data, including the pages that ranked for the query, and the impressions, clicks, and click-through rate at each position the site ranked.

Country and Property-Specific Data

Use the filters to drill into what queries brought traffic from Google properties (web, video, images, mobile web, and smartphones) and from specific countries.

How the Numbers Are Aggregated

As I explained in a previous post, the numbers can be tricky and it’s important to understand what data you’re really looking at. These reports now list the top 2,000 queries that brought traffic to the site for the selected time period. That means that if a query wasn’t one of the top 2,000 for any days in the selected range, data won’t be reported for it. In the example below, the time period is 30 days, but only 6 of those days have data reported for the query (as illustrated by the dots in the graph).

Generating Data

When generating or downloading the data, keep in mind the following:

• The default ending data in the user interface display is today’s date, but reporting is typically 2-3 days behind so check the last date reported in the graph by hovering over the last dot. The default starting date of the range is 30 days before the end date. And because this end date is generally three days ahead of what’s actually reported, the actual date range generally shown by default is 27 days. Make sure you adjust the dates before analyzing data or comparing it to other time periods.
• Because the default shown date range is 27 days, the download available from the Python script is also 27 days.

Last week, the Bing Webmaster blog published a post about how Bing handles rel=”next” and rel=”prev” attributes. On the surface, it seemed as though Bing was announcing that it now supported these tags in the same way Google does. Last September, Google announced support of the rel=”prev” and rel=”next” attributes to designate paginated content, which enables site owners to cluster multiple pages of content into single entities so that indexing and other values can be consolidated.

However, the conclusion of the post included the following, which didn’t align with Google’s treatment of these tags:

“Implementing these rel=”next” and rel=”prev” link elements doesn’t trigger a new visual treatment for your pages on our search result pages. It does, however, allow us to more comprehensively understand and index your content”.

I talked to Bing’s Duane Forrester to clarify how Bing treats these tags. He confirmed that Bing does not use these tags to consolidate pages of a series into a single entity. Rather, Bing may use this markup in two ways:to aid discovery of pages and to enhance the search results display (possibly with links to the next and previous page, for instance). Although it wasn’t clear if either was happening yet. He said:

“We may use our newly gained knowledge on your site’s structure to provide easy access to other sections of the paginated or sequenced content from our results pages in the future. In addition, webmasters implementing these elements may benefit from more comprehensive indexing over time as we apply our newly gained knowledge of a site’s structure to our indexing heuristics.”

So what’s a site owner with paginated content to do? The best bet is to implement the tags for Google’s benefit and as a bonus, get potential additional crawling or display improvements from Bing. If pagination of your content is a significant problem in Bing (which may be the case particularly for some large e-commerce sites), consider blocking excessive pages with a Bing-specific robots.txt directive or noindex meta tag.

“For those who worry that 1000 error details plus a total aggregate count will not be enough, we’re considering adding programmatic access (an API) to allow you to download every last error you have, so please give us feedback if you need more.”

And led me to believe that the current API would only provide access to the same data available from the downloads from the UI. But in any case, up to 100,000 URLs for each error and the details of most of what has gone missing is in fact available through the API now, so rejoice!

The data is a little tricky to get to and the specifics of what’s available varies based on how you retrieve it.  Two different types of files are available that provide detail about crawl errors:

• A download of eight CSV files, one of which is a list of all crawl errors
• A crawl errors feed, which enables you to programatically fetch 25 errors at a time

(Thanks to Ryan Jones and Ryan Smith for help in tracking these details down.)

What this means is that different slices of data are available in four ways:

• User interface display
• User interface-based CSV download
• API-based download
• API-based feed

CSV Download

Eight CSV files are available through the API (you can download them all for a single site or for all sites in your account at once as well as just a specific CSV and a specific date range), but this support is not built into most of the available client libraries. You’ll need to build it in yourself or use the PHP client library (which seems to be the only one that has support built in). The CSV files are:

• Top Pages
• Top Queries
• Crawl Errors
• Content Errors
• Content Keywords
• Internal Links
• External Links
• Social Activity

For the topic at hand, let’s dive into the crawl errors CSV. It contains the following data:

• Up to 100,000 URLs for each type of error (rather than the 1,000 maximum available through the download link in the UI)
• The full list of URLs blocked by robots.txt (which is no longer available at all in the UI)
• Specifics of “not followed” errors (the UI reports only the status code returned by the URL, while the CSV includes what the actual problem was, such as “too many redirects”)
• Specifics site-wide server errors (the UI no longer lists the specific URLs that returned the error or the specific error)
• Specifics about “soft 404s” (the UI doesn’t include the detail of the type of soft 404)

This file does not include details on crawl error sources (but that is available through the crawl errors feed, described below).

Crawl Errors Feed

It appears that the crawl errors feed request code is built into the Java and Objective C client libraries, but you’ll have to write your own code to request this if you’re using a different library. You can fetch 25 errors at a time and programmatically loop through them all. The information returned is in the following format:

<atom:entry>
  <atom:id>id</atom:id>
  <wt:crawl-type>web-crawl</wt:crawl-type>
  <wt:issue-type>http-error</wt:issue-type>
  <wt:url>http://example.com/dir/</wt:url>
  <wt:detail>4xx Error</wt:detail>
  <wt:linked-from>http://example.com</wt:linked-from>
  <wt:date-detected>2008-11-17T01:06:10.000
  </wt:date-detected>
</atom:entry>

How the Data Differs

Let’s take a look at some real examples at how what you get in this file differs from the UI and UI-based download.

Number of Errors Shown

The UI shows the total count over time, but only lists up to 1,000 URLs for each error. The API-based CSV contains up to 100,000 URLs for each type of error. However, it gives you only the current snapshot of errors and doesn’t provide a total count of each error (if more than 100,000 exist for any given one). This is the same information you can get from the API-based crawl errors feed. The UI-based CSV also only shows you the current snapshot of errors and lists only up to 1,000 URLs (the same as the UI).

For searchengineland.com’s “not found” errors:

• The UI shows that Google encountered 4,981 URLs (down from around 8,000 in December) and displays 1,000 of them, along with the corresponding response code (and the list of sources via a popup):
• The UI-based CSV lists 1,238 of them, along with the corresponding response code:
• The API-based CSV lists 2,867 of them, along with the corresponding response code and the number of incoming links (but not the sources):
• The API-based feed presumably would provide all 4,981 URLs (in 25 URL increments), the response code, and the list of sources.

I checked a site that the UI indicated had more than 100,000 of a particular error (257,065) and found that the corresponding API-based CSV file listed 100,005 of those. The UI-based CSV listed 1,999.

Error Sources

The UI shows error sources, but only at the individual URL level. Click a URL, then click the Linked From tab to see the list. You can’t download this list from the UI in aggregate or for an individual URL.

The API-based CSV indicates the number of sources linking to each URL (which can be useful as you can tackle issues with URLs that have a lot of links first), but doesn’t list what those sources are.

The API-based feed provides details on the source for each URL. Below is an example from searchengineland.com:

<atom:entry>
<atom:id>https://www.google.com/webmasters/tools/feeds/

http%3A%2F%2Fsearchengineland.com%2F/crawlissues/27</atom:id>
<atom:updated>2012-03-19T17:18:18.907Z</atom:updated>
<atom:category scheme='http://schemas.google.com/g/2005#kind'
 term='http://schemas.google.com/webmasters/tools/
2007#crawl_issue_entry'/>
<atom:title type='text'>Crawl Issue</atom:title>
<atom:link rel='self' type='application/atom+xml'
href='https://www.google.com/webmasters/tools/feeds/
http%3A%2F%2Fsearchengineland.com%2F/crawlissues/27'/>
<wt:crawl-type xmlns:wt='http://schemas.google.com/
webmasters/tools/2007'>web-crawl</wt:crawl-type>
<wt:issue-type xmlns:wt='http://schemas.google.com/webmasters/
tools/2007'>not-found</wt:issue-type>
<wt:url xmlns:wt='http://schemas.google.com/webmasters/
tools/2007'>http://searchengineland.com/10-optimization-
secrets-to-drive-more-mobile-traffic-from-facebook-114316/
www.linkedin.com/in/brianklais</wt:url>
<wt:date-detected xmlns:wt='http://schemas.google.com/
webmasters/tools/2007'>2012-03-17T05:58:35.000</wt:date-detected>
<wt:detail xmlns:wt='http://schemas.google.com/webmasters/tools/2007'>
404 (Not found)</wt:detail>
<wt:linked-from xmlns:wt= 'http://schemas.google.com/webmasters/ tools/2007'>htp://searchengineland.com/10-optimization-secrets-to- drive-more-mobile- traffic-from-facebook-114316/comment-page-1 </wt:linked-from>
</atom:entry>

Date of Latest Crawl

As far as I can tell, this detail is available only in the UI (once you click on a URL):

In Sitemaps

This level of detail also seems to be only available in the UI (once you click on an individual URL and then the In Sitemaps tab). This information is also available in aggregate from the Sitemaps section of the UI (but not as a download — either from the UI or from the API).

Not Followed Errors

The UI shows the URL’s response code (such as 301), which as I noted in my earlier article, is somewhat misleading and not that useful for investigating the error. The report can be misread to mean that 301 response codes are errors. What this report actually provides is a list of URLs that returned either a 301 or 302 response code that Googlebot couldn’t follow due to a problem with the redirect.

The UI-based CSV provides similar data (although not the additional details available in the UI when you click the URL).

The API-based CSV lists “redirect error” for these specific URLs.

That particular error isn’t any more helpful than the response code, but in some cases, this file lists the most specific issue (this detail is one of the pieces of data that used to display in the UI). Possible values include:

For instance, here’s a (slightly obscured) example of a URL with a “redirect URL too long” error:

http://www.example.org/A%20Category%20Page%0A/Www.Sample.Net

%0A/Topic%20Topic1%0A/A%20Category%20Page%20Topic%20Topic1 %0A/Topic%20Topic3%0A/Excercise%0A/Smoothies%0A/Topic2ix%0A/ Topic%20Trainer%0A/Topic3%20Programs%20%0A/Topic%20Assessments %20%0A/Individual%20Or%20Group%20Topic3%20Topic1%20%0A/Topic4 %20Guidance%20%0A/Category2%20Loss%20Supervision%20%0A/Category2 %20Loss%0A/Support%20And%20Motivation%20%0A/Topic9%20Specific %20Category4%20%0A/Cardiovascular%20Category4%20%0A/Topic9%20 Specific%20Category4%20%20%20%0A/Home%20Gym%20Selection%0A/ Home%20Gym%20Use%0A/Topic12%0A/Topic10%20Topic15%20Topic12%0A/ Workout%20Routines%0A/Category2%20Topic1%0A/Topic3%0A/Topic9%0A/ Topic11%20Topic12%0A/Lean%0A/Lean%20Category3%0A/Nutrition%0A/ Gift%20Topic13s%0A/Topic3%20And%20Wellness%0A/Category5%0A/Topic3 %20Programs%20For%20All%20Ages%20%0A/Topic%20Assessments%20%0A/ Individual%20Or%20Group%20Topic1%20%0A/Topic4%20Guidance%20%0A/ Category2%20Loss%20Supervision%20%0A/Support%20And%20Motivation %20%20%0A/Cardio%20%0A/Topic9%20Specific%20Category4%0A/Category2 %20Topic1%0A/Category2%20Topic14%20Routines%0A/Premier%20Topic%20 Trainer%0A/Category2%20Loss%20Help%0A/Safe%20Category2%20Loss%0A/ Wellness%0A/Nutrition%0A/Category3%20Topic3%0A/Exercise%20Health%0A/ Category2%20Loss%20Product%0A/Category6ing%0A/Topic3%20Equipment%0A/ Category2%20Loss%20Supplement%0A/Loose%20Category2%0A/Health%0A/ Healthy%20Category6%0A/Fat%20Loss%0A/Category6%20Plan%0A/Online%20 Topic3%20Program%0A/Topic3%20Center%0A/Gym%20Exercise%0A/Fast%20 Category2%20Loss%0A/Lose%20Category2%20Fast%0A/6%20Topic15%20 Topic12%0A/Home%20Gym%20Equipment%0A/Gym%0A/Topic3%20Course%0A/ Topic3%20Exercise%0A/Gym%20Equipment%0A/Topic3%20Topic1%20Program%0A/ Gym%0A/Exercise%20Program%0A/Category2%20Loss%20Category6%0A/Topic%20 Topic3%0A/Exercise%0A/Sport%20And%20Topic3%0A/Topic3%20Class%0A/Topic %20Topic3%20Trainer%0A/Topic10%20Topic15%20Exercise%0A/Health%20And%20 Topic3%0A/Topic3%20Australia%0A/Category2%20Loss%20Program%0A/Topic3%20 Trainer%0A/Topic%20Topic1%20Topic16%0A/Topic3%0A/Kick%20Topic17%0A/ 6%20Topic15%0A/Category2%20Topic1%0A/Topic%20Trainers%0A/Category2%20 Loss%0A/Topic%20Trainer%0A/Home%20Gym%0A/Topic%20Topic1%0A/Category6 %0A/Quick%20Category2%20Loss%20%0A/Category2%20Loss%20Plan%0A/ Category7%20Category2%20Loss%0A/Easy%20Category2%20Loss%0A/Category2 %20Loss%20Tip%0A/Healthy%20Category2%20Loss%0A/Rapid%20Category2%20 Loss%0A/Topic8%20135%0A/Category2%201

An “invalid” redirect brought me to this page (which was a 302 to a 404):

Empty redirects are those with no location information.

The API-based feed also provides these specifics as part of the feed.

Soft 404s

Gone missing from both the UI and UI-based CSV are the specifics of soft 404s.

The API-based CSV still lists these details (as does the crawl errors feed), such as:

• 404-like content (the page returns a 200 response code, but seems to contain contain from an error page)
• Redirect to an error page

Site-Wide (Server) Errors

With the UI overhaul, Google shows the number of site-based (vs. URL based) errors encountered over time, but not the specific URLs that triggered the errors, and has simplified the error messaging into three types: DNS, server connectivity, and robots.txt fetch. There is no corresponding download. (Google says when they encounter these types of errors, it typically means they’ll receive them for any URL on the site, since the problem is at a lower level.)

There’s some interesting data to be gleaned here that’s not apparent. For instance, if you hover over the dots, you see the total number of Googlebot fetch requests per day. (Unfortunately, you can only see this graph if you Google encounters at least one site error during the reported time period and there’s no way to see this number other than hovering.)  For one site I looked at, the number of URLs crawled averaged around 250,000 per day. Then one day, 2% of requests returned a DNS error (2,866 of 152,528 requests). The following day, Googlebot made only 128 requests (all crawled successfully) and only 71 the day after that. This doesn’t match up exactly what the crawl stats report shows:

Another site went from an average of 100,000 a day to 362 a day after 1% of requests returned a DNS error.

With the API-based CSV file and API-based crawl errors feed, you get:

• The URL that triggered the error
• The specifics of the error

DNS Error

Server Connectivity

Robots.txt Fetch

Pages Blocked with Robots.txt

This report is gone entirely from the UI but it’s still available (up to 100,000 URLs) from the API (both the CSV and the feed).

All in all, I’m happy to learn that most of this information is still available. It’s somewhat cumbersome and confusing that what’s available isn’t consistent across delivery mechanisms and figuring out how to access the API-based data isn’t all that straightforward. But for those power users (like me) who process this data programmatically, that Google is continuing to provide this information is great news.

How the Pagination Attributes Work

The pagination attributes can be used for any set of content that spans multiple pages. Typical scenarios include multi-page articles, product listings, and forum discussions.  Simply use the rel=next and rel=prev attributes to link all pages in a series together. For the following set of pages:

• www.site.com/products?page=1
• www.site.com/products?page=2
• www.site.com/products?page=3

The pagination attributes would be as follows:

Page 1:

<link rel=”next” href=”http://www.site.com/products?page=2″>

Page 2:

<link rel=”prev” href=”http://www.site.com/products?page=1″>

<link rel=”next” href=”http://www.site.com/products?page=3″>

Page 3:

<link rel=”prev” href=”http:// site.com/products?page=2″>

When To Use Pagination Attributes Instead of Canonical Attributes

Some sites are set up to use the canonical attributes to point all pages in a series to page one. As Maile points out in the video, this isn’t the correct use of the canonicalization tag (in part because Google only indexes the content on the canonical page, so any content from the rest of the pages in the series would be ignored).

If the paginated content is a subset of the canonical page (such as when you have a view all version or a filtered result set) or is identical (such as when the sort order changes the display but not the content), then use the canonical attribute instead of the pagination attributes.

General Best Practices

Use Absolute URLs

The href values can be absolute or relative (the original version of this article said they had to be absolute, but that version was incorrect). But using absolute URLs is a best practice, both to combat scrapers and in case URLs are duplicated accidentally across directories or subdomains.

The Chain Can’t Be Broken

The rel=”next” and rel=”prev” values must match. If they don’t, the chain is broken. For instance, for the following pages:

• www.site.com/products?page=1
• www.site.com/products?page=2

The rel=”next” attribute for page=1 must point to page=2 and the rel=”prev” attribute for page=2 must point to page=1.

The pagination attributes can only link together URLs with matching parameters. For instance, the following URLs aren’t considered part of the same series, as the second URL would break the chain:

• www.site.com/products?page=1
• www.site.com/products?page=2&referrer=twitter
• www.site.com/products?page=3

This means that ideally, you should dynamically insert the pagination values based on the fetched URL. In the case of the above example,when Googlebot fetches the page as:

www.site.com/products?page=2&referrer=twitter

The pagination values should be (dynamically inserted as):

<link rel=”prev” href=”http:// www.site.com/products?page=1&referrer=twitter”>

<link rel=”next” href=”http://www.site.com/products?page=3&referrer=twitter”>

Each page can be in only one pagination chain

• A page can’t contain multiple rel=”next” attributes.
• Multiple pages can’t have the same rel=”prev”.
• A page that contains a rel=”canonical” attribute to another page can’t be part of the canonical URLs paginated series. It must be paginated with the URLs that match it (and then Google will use the canonical attributes to consolidate each page accordingly). See more on how this works at the end of this article.

Advanced Techniques

Product listings in particular often have additional complexity, such as sort orders and filtered navigation. It’s best to start with the simplest paginated series and then make canonicalization and pagination decisions for each level of complexity. You can’t specify one view/filter as canonical and point all other versions of the paginated series to that default set.

Viewing and Sorting Options

If a set of product listings has multiple view options and those listings span multiple pages, you have to create a pagination set for each view option separately (since the pages aren’t subsets of a default version). For instance, if you provide options to view 20 products at a time or 100 products at a time and to sort by newest, price, and ratings, you would need to implement a separate paginated series for each view option. As you might imagine, this could cause you to end up with a large number of paginated series’. Shown below are just two of the many possible:

Filtering (AKA Faceted Navigation)

Things get even more complex when you introduce filters.

• If the filtered view is a subset of a single non-filtered page (perhaps the view=100 option), you can use the canonical attribute to point the filtered page to the non-filtered one. However, if the filtered view results in paginated content, this may not be viable (as each page may not be a subset of what you would like to point to as canonical).
• If you want the filtered view to rank separately from the default view, you would create a paginated series for each filtered category. You would need to also paginate all of the various sort and view options separately. Take REI.com as an example. The site has a Snowboards section that would likely be paginated and (hopefully) rank for [snowboards] queries. But a filtered view of just Women’s Snowboards, while a subset of the snowboards content would likely be paginated separately in order to rank for [women's snowboards] queries. (On the other hand, REI probably doesn’t want the filter by size version to rank separately, so that variation could be canonicalized.)

Using the Canonical Attribute in Conjunction with the Pagination Attributes

Each URL that’s paginated separately should also contain a canonical attribute. In the case of the earlier example with the optional referrer parameter, for instance, Google will first consolidate the default paginated series separately, and then the paginated series that contained the referrer=twitter parameter separately, but then use the canonical attribute of the pages to further consolidate the pages to the default version. That means that the URL of:

www.site.com/products?page=2&referrer=twitter

Would end up with the following markup:

<link rel=”canonical” href=”http://www. site.com/products?page=2″>

<link rel=”prev” href=”http://www. site.com/products?page=1&referrer=twitter”>

<link rel=”next” href=”http:// www.site.com/products?page=3&referrer=twitter”>

Too Confusing To Implement?

I admit, all of this sounds confusing. But it’s not so bad if you take things step by step:

1. What is the canonical version of each URL? Add the canonical attribute to the pages.
2. What is the default view for each paginated series? Add the pagination attributes to these pages.
3. What views/filters are subsets of broader views? Add the canonical attribute to these pages to point to those broader views.
4. What views/filters are not subsets of broader views or you want to rank separately? Add separate pagination attributes to these pages to make each a separate series.

Update 3/17/12:After talking with Google, I’ve learned that most of what I was disappointed to find had been removed and that I feel is useful detail for power users is in fact still available through the API! I’ve dug into the details and have written up my findings. I’ve also updated this story with additional details from Google:

• Access denied errors include 401, 403, and 407. That some of these were showing up as “other” was a bug that has since been fixed.
• Not followed errors are indeed URLs returned either a 301 or 302 and Googlebot had trouble crawling that redirect due to an issue.

So what’s changed?

Site vs. URL Errors

Crawl errors have been organized into two categories: site errors and URL errors. Site errors are those which are likely site-wide, as opposed to URL-specific.

Site errors are categorized as:

• DNS – These errors include things like DNS lookup timeout, domain name not found, and DNS error. (Although these specifics are no longer listed, as described more below.)
• Server Connectivity – The errors include things like network unreachable, no response, connection refused, and connection reset. (These specifics are also no longer listed.)
• Robots.txt Fetch – These errors are specific to the robots.txt file. If Googlebot receives a server error when trying to access this file, they have no way of knowing if a robots.txt file exists, and if so, what pages it blocks, so they stop the crawl until they no longer get an error when attempting to fetch it.

Clay’s a friend of mine and I’ve read his book (it’s quite good, by the way), so I knew both that he doesn’t really think that SEO is killing America and that he’s unaware just how much we’re all over this particular linkbait-y title.

And indeed his talk was not about how SEO is killing America. Instead, it was about two things:

• As a culture, we want to be entertained and told that we are right. It’s much easier for news organizations to sell news that reaffirms our opinions than news that educates and challenges us.
• News organizations need page views, so policies such as the “AOL Way” may sacrifice investigative journalism at the altar of popular search queries.

Clay’s talk (and his book) are mostly about the former, but my interest is in the latter. In his talk, Clay noted that we broadcast what we want by way of our searches and clicks. In turn, others see the content we’ve made popular in “most read” modules on news sites and content creators write more articles on popular topics based on search volume. The danger is that we don’t always seek out stuff that’s good for us and the more we look for what’s more fun to consume, the more that’s all that’s available.

He cited the “AOL Way” and the practice of using search data to determine traffic potential of topics and to decide what to write more about as an example of how the media’s focus on SEO may be an obstacle to the best possible news coverage.

For me, this argument is another variation of the “SEO is spam” argument. Spam is spam and lumping it in with solid search engine optimization processes doesn’t make it SEO. Creating content simply based on popular search terms isn’t SEO either. In my book Marketing in the Age of Google, I addressed this issue at length and wrote about how tactics of spammers were mislabeled as tactics of SEO, but that it may be too late to reclaim the term. There, I wrote:

Integrating a search acquisition strategy into a more comprehensive business strategy includes:

• Using search data to build a comprehensive and effective product and content strategy.
• Understanding searcher behavior and building searcher personas that maximize customer satisfaction and conversion.
• Realizing the customer acquisition funnel often begins with the search box, not your web site.
• Integrating organic search with other marketing efforts.
• Ensuring technical architecture o the site can be properly crawled  and indexed by search engines so that it can be visible to searchers.

I have explored the search data issue in depth as it relates to journalism during my National Press Club workshops. At least three components are involved:

• Search data is valuable for learning what your audience is interested in to help ensure you meet their needs.
• It’s important for content creators (including journalists) to understand how to connect with searchers in order to gain maximum visibility.
• Investigative journalism is vital, and search may not be the best initial channel for reaching readers.

 Using Search Data

As with nearly everything else in life, you can use search data for good or for evil. Take the Super Bowl start time, for instance. In 2011, the Huffington Post famously spammed the hell out of Google by creating an article that basically just repeated every variation of related search query. Not only did this article contain little useful information, but one wonders if Super Bowl viewers are really a key target audience for a supposed news site or if the point was more about page views that keeping the public informed on the issues of the day.

But with the latest Super Bowl in 2012, the NFL created a page specifically for those seeking out information about the game schedule. Although they were using the same search data, not only was the page useful, but it addressed the NFL’s target audience. The point was obviously not about simply page views but to engage with viewers and get them to interact with additional content on the site. I talked to John Cole, who recently joined NFL.com to head up search and social media and is  responsible for this new tactic at NFL.com. He told me that user testing found that their target users found the information they were looking for regarding the game schedule much more quickly with the details they added to the pages. To me, this is a perfect use of search data: find out what your audience is looking for and answer their questions (making them happy and keeping them engaged with your brand).

The attempt of to simply maximize page views by creating pages about popular topics is not caused by the availability of search data. This type of reporting has existed since the beginning of time and the online medium simply provides new opportunities for creativity. For instance, when reading an article a few days ago, I came across the following set of headlines:

Why indeed did M.I.A flip the bird during the Super Bowl? When I clicked through to the page, I first encountered this:

Entertainment Weekly certainly is taking a page from HuffPo’s playbook by filing this story under as many keywords as possible. But what about the story itself? Do we find out why she did it?

Not exactly.

Being Visible To Your Target Audience

In the olden days of yore, the printed newspaper arrived at one’s door, and one flipped through the pages and skimmed through the headlines while drinking one’s morning coffee. Wearing a corset (or top hat depending on one’s fashion leanings). But things have changed. Now, when we want to news, we either go to an online source such as Google News or we search for exactly what we want to know. You can see this by checking search volume for  just about any news item. See for instance, search volume for [healthcare reform] queries:

Not only should journalists use search data to make sure they’re answering all of the questions their readers have about a particular topic, but they should make sure they’re using the language of their readers so that when those readers seek out content, the news stories appear. (You can see how simply a spelling change can make all the difference in the world with different spelling guidelines for “Gaddafi”). It’s not spamming or killing America to make sure that your headline contains descriptive words that match how readers are searching for stories.

This doesn’t only help news stories appear for the right searches but helps click through on those headlines on news sites and aggregators.

What About Investigative Journalism

Investigative journalism is trickier. No one is searching for information about the topic at hand until the story breaks, but how to get the news out there in the first place?  Certainly, this type of journalism is tougher to disseminate.  It was easier in yonder days of yore with the printed paper and the doorstep and the corsets and the like.  It can seem like a lot less trouble to just write stories that you already know people are searching for information about. I asked Clay how he recommended journalists go about getting readers for stories no one was searching for and he told me:

“I think the best asset an investigative journalist can have is a strong social network. But let’s not also forget that journalists usually come with a distribution point baked in. People still do read the paper. People go to nytimes.com.”

And he pointed out that these stories can drive search interest. What the media chooses to cover and the words they use to describe events have direct impact on what people search for and how they search. Once a breaking story hits, people do in fact begin searching for more information about it.

So perhaps, in fact, SEO isn’t killing America but can instead keep America informed.

Commercials Drive Searches

Since the 2009 Super Bowl, I’ve monitored how the ads influence search interest, and every year, the trend has been the same. As people watch the Super Bowl, they search for everything they’re watching: teams, players, performers, and of course, commercials. The trend continues the day after the game as people talk about the commercials and turn to Google (and Bing) to watch them again. Take a look at the spiking searches for February 7th, the day after the game according to Google Trends:

Nearly every search is Super Bowl related, and searchers are clearly seeking out the ads. As you can see from search #8, commercials often cause people to search for the brands directly. Google Insights for Search shows that brands that advertised saw significant search spikes on Sunday. See for instance, the search volume for [bud light platinum].

They seemed to have really liked those ads in Iowa.

Google reported that searches for [super bowl ads] were 122 times higher this week and that the big search winners were Acura, GoDaddy, and M&Ms.

Where Are Advertisers Sending Potential Customers?

As I do every year, I took note of what advertisers included in the commercial. Did they include a web site URL? A Facebook page? Did they seem to even be aware of this crazy new thing called the internet? And then I looked at the advertisers’ search visibility. I was looking for the following flow:

Last year, many only paid attention to a flow like this:

I understand that Super Bowl commercials are about branding, not necessarily instant purchases, and I realize other positive outcomes exist (discussions on social media and the like). I’m just saying that if someone is searching for you, you may as well show up. And if you’ve gotten potential customers to view your commercial, you may as well make it easy for them to view more information about your products.

This year, many advertisers simply included their domain name in the ad (33 of the 53 advertisers I tracked did this). This approach can help cut out the search step, although as the response to the Dockers ad during the 2010 Super Bowl showed, advertising a URL causes people to, well, search for the URL. So you can’t always cut out the search step, no matter how hard you try.

Last year’s Super Bowl ads were all about Facebook fan pages (that often were impossible to find; don’t say “find us on Facebook” unless that’s an achievable task). This year, only fourof the ads included a nod to Facebook and all used actual URLs. Pepsi Max even went with an easy to remember redirect to Facebook: pepsimax.com/facebook.

Four commercials advertised Twitter hashtags (last year was the first year for this, and then it was mostly only for movie trailers). I was astonished to find that when a hashtag was included in a commercial, people instantly started using it to tweet about the commercial and the hashtag began trending. (As you can see, even the bands with songs in the commercials started trending.)

Of course, there’s a risk in this strategy. Things may go really well, as Audi found with #SoLongVampires, or very awry as Bud Light found with #MAKEITPLATINUM. (Did people really even use the same capitalization in the hashtag as was used in the commercial? Amazing.)

What began trending on Twitter also tended to show search spikes. For instance, take a look at searches for [echo and the bunnymen]:

So what we talk about, we also search for.

The Future is… QR Codes?

It may have seemed like GoDaddy used the same tired formula as always in their ads (although, apparently sex does sell, so I can’t knock sticking with something that works), but in fact, they tried something new this year: including a QR code in the ad.

The online version of the commercial includes the QR during the entire length of the ad, but when aired during the Super Bowl, it appeared only briefly at the end, so I’m not sure if  anyone managed to pull up the QR code reader on their mobile phone, rush to the TV, and scan it before it disappeared from the screen. Including it in the online version seems even more nonsensical though, as the idea seems to be that you’re watching the ad on your computer, see the QR code, scan it with your phone, and are brought to the godaddy.com site on your phone. I would guess that including a link to the web site in the commercial so that you can simply click and access the web site on your computer would make entering your credit card information for all those domain names quite a bit easier.

Scoring Search Visibility

So how did advertisers do in search? It’s difficult to come up with exact search coverage percentages. For instance, if a brand advertised multiple products and ranked well in search results for one product but not the other does the tick mark for that brand go in the yes or no column for search visibility? What if the product showed up for its name but not for its tagline?

For the purposes of the stats below, I used the following guidelines:

• I counted each brand once, even if they aired ads for multiple products
• If they ranked organically for at least one of brand, product, or tagline queries, I put a yes in the organic search column
• If they had a paid search ad for at least one of brand, product, or tagline queries, I put a yes in the paid search column

In a follow up column, I’ll point out some interesting choices, but for now, let’s just look at how well advertisers thought out web sites, search, and social media.

Of the 53 brands I tracked:

• 33 ended the ad with a URL to the brand site, 4 went with a Twitter hashtag, and 4 sent viewers to Facebook.
• 44 bought a paid search ad
• 51 ranked organically for the brand name (although far fewer ranked for the promoted taglines or hashtags)

Chrysler and YouTube

Last year, Chrysler’s Eminem ad was one of the most popular commercials of the game. I found it odd at the time that although they designed their site’s home page to tie in quite well to the vibe of that ad, they bought search ads to the commercial on YouTube. I felt they lost an opportunity to further interact with potential customers and lost some control of the experience (related videos could easily be to competitors, for instance). Their flow looked like this:

That’s not a bad outcome, but I thought that if they had used paid search to drive visitors to the commercial on their site, they might have been able to better leveraged the opportunity. This year, Chrysler once again had a much-talked-about ad, and they decided to mix things up a bit.

For [chrysler]-related searches, the paid search ad points at their home page, which is a great tie in to the commercial. But for other searches, they’ve once again chosen to promote YouTube.

This time, the YouTube link makes a lot more sense as it’s to the channel, so there are no competitor links and the entire page is focused on getting votes for the YouTube AdBlitz, engaging socially, and even includes an ad for the car featured in the commercial. All in all, I fully support this approach. They keep the branded searches pointing at their home page (after all, not everyone searching for the brand is searching for the commercial), which is tightly-integrated with the campaign, and they send those looking for the commercial to a page designed to specifically engage with them.  What a difference a year makes.

2012 Paid Search Ad to YouTube:

2011 Paid Search Ad to YouTube:

Acura NSX vs. Bud Light Platinum

We’ve already seen that the #makeitplatinum hashtag strategy both worked and didn’t work for Bud Light (they definitely got it trending, but for perhaps the wrong reasons). What about organic search visibility? Sadly, the brand web site doesn’t appear at all in Google for searches for [bud light platinum] (although they have bought a paid search ad to the YouTube page).

Acura NSX, on the other hand (which was a spiking search on Monday), does an excellent job with organic search, taking the top spot with a page devoted to it. (Although including the commercial on the page would have been a good idea.)

Overall, I felt brands did a much better job of keeping things simple and driving viewers to interesting, relevant pages that engaged them. Watch for my next post in the coming days for some specifics on what went right and spectacularly wrong.

(Stock image via Shutterstock.com. Used under license.)

Related:

• Super Bowl 2012: What Time Does It Start?
• Super Bowl 2011: Commercials and Search Visibility
• Super Bowl 2010: Commercials and Search Visibility
• Super Bowl 2009: Commercials and Search Visibility

Amazing because every year, football fans flock to search engines searching for the start time, and until now, organizations like the NFL, the playing teams, and the broadcasting station didn’t show up at all in search results because none of their sites answered the question. Seem crazy?

• 2009  Results: In 2009, start-time related searches were among the most popular the morning of the game, but neither the NFL nor NBC were anywhere to be found.
• 2010 Results: In 2010, both nfl.com and cbs.com had significant technical infrastructure issues that kept search engines from crawling and indexing the content. Again, the search results were sad and this time, full of spammers trying to capitalize on the search volume.
• 2011 Results: In 2011, problems continued. But news organizations jumped in, and the Huffington Post in particular ranked well for its article that simply listed all of the various ways people were searching for the Super Bowl start time. (That article was later “edited for clarity”).

This year, things are finally getting better. Even the Huffington Post, while still getting every variation of spelling and tagging in the article for maximum search coverage (“For starters, it’s two words, not one. “Superbowl” is an incorrect spelling.”), has filled out their article a bit with actual information.

The results could still be better. While [Super Bowl start time] has overall higher search volume than [Super Bowl kick off time], the latter is the top search this morning, and NFL.com only ranks for the former (HuffPo does quite well with the latter). Superbowl.com, which redirects to the NFL site, ranks, but as I mentioned in earlier years, this domain 302 redirects to nfl.com. A 301 instead would consolidate the domains (including value signals such as links), which might cause the target URL to do better overall in relevant searches. But still, compared to earlier years, I’d call these results a win for the NFL.

Sadly, NBC, the Giants, and the Patriots, and TV Guide all fail to appear in results once again. Even though both Google Insights for Search and my articles over the years should have prepared them for this year’s search interest.

Why should these sites care about showing up for these searches? They’ve invested substantially in site content and those seeking out the game start time are a perfect audience for that content. Searchers would click for the start time and stay for the fan jam videos and view the ads.

Of  course, Super Bowl viewers will see lots of ads anyway today. But that’s a topic for the next article.

Related:

• 2009 Super Bowl Start Time
• 2010 Super Bowl Start Time
• 2011 Super Bowl Start Time
• 2009 Super Bowl Commercials
• 2010 Super Bowl Commercials
• 2011 Super Bowl Commercials

Search Queries: Download Chart Data

Search query chart data downloads provide access to information not easily available before. When you view search query data, the chart in the user interface shows impression and click data per day, but in the past, this detail has only been available if you hover over a dot in the chart.

Now, you can access this data by clicking the Download Chart Data  button below the Top Queries and Top URLs reports. (You can use filtering options to drill into specific data or date ranges).

The data downloads to a CSV file as shown below:

When you click on a specific query in the Top Queries report page, you see a similar chart, but its data is not available for download. In the example below, clicking on [track santa claus] query in the Top Queries report brings up the query details for that single query, and you can download the page and position information, but not the chart data. (Note that a display bug seems to be preventing the buttons on this page from displaying in Internet Explorer.)

You can filter the Top Queries report and download the impression and click data from there (although you may not be able to filter to a single query). In the example below, a filtered report for [track santa claus] resulted in 16 queries.

The chart data in this case can be particularly useful due to how Google reports trended data. Watch for a separate post detailing how this works, but in short, Google reports on the top 1,000 queries per day. For reports that span multiple days, such as the example above, impression and click data is only included if the query was in the top 1,000 for each day. This can result in misleading totals, since the impressions and clicks aren’t totals for the entire data range, but only totals from the days in that range that the query was in the top 1,000. Confusing? Totally understandable.

Let’s look at the example above. I’ve filtered the Top Queries report to only show queries that include [track santa claus] for a 31 day period and the report shows:

• 16 queries
• 170,000 impressions
• 3,000 clicks

Does this mean the site got traffic for only 16 queries that contain the words [track santa claus]? No, it means that only 16 queries made it into the top 1,000 at least one day during the date range. Based on that foundation, you might assume that those 16 queries resulted in 170,000 impressions for the site and brought in 3,000 clicks for the 31 day period. But that assumption would be wrong. (This is where things can really start to get confusing.)

The Search Queries report only reports data (at all) for those days that a query is in the top 1,000. And the totals are the sum of the counts on those days only — not the entire date range. This level of detail has always been available (sort of) by way of the dots on the charts. Looking more closely at our [track santa claus] example, notice that several days in the chart don’t have dots.

The days without dots are the days that the query (or set of queries) didn’t make the top 1,000 and therefore have no data reported. Until now, the only way to make some sense of the data was to count the dots to better understand the summary data. Now, when you download the chart data, you can much more easily understand what the visualization represents:

With the CSV, it’s much more apparent that although the specified date range is 12/18/11 – 1/17/12, the totals are for 11 days, not 31. These queries were in the top 1,000 on 12/27, but not on the 28th or 29th. Does this mean that on the 28th, the impressions and clicks were lower than on the 27th? Maybe. Or it could mean that impressions and clicks stayed the same but another query spiked in volume and bumped it from the top 1,000.

User Interface Improvements

One nice change is the ability to view more data on a page. Most pages, including the dashboard page listing all sites in the account, now include an option for the number of rows to show.

You can now view up to 500 rows at once, which makes overall account management and data viewing much easier.

I had initially thought that more link data might be available as well, but after talking with Google, I realize that the user interface improvements have simply made the link reports more easily visible.

In particular, the detailed chart data available for download provides a significant improvement as it brings a much more accurate understanding of what the data actually is. Watch for my more detailed post on the search query data shortly.