I’ve written and presented often on pagination for SEO. Why so much attention on this subject?

The reason is simple: it can be a big, hairy deal for sites. It’s right up there with faceted navigation as one of the most problematic crawling and indexing issues for large-scale SEO. It’s a tactic (actually a set of tactics) that our teams are continually evolving, testing, and refining.

So it was “double prizes” when Google announced the HTML 5 element rel next/prev for pagination.

3 Overall Tactics For SEO Pagination

There are three primary tactics that we use for SEO pagination:

• Classic Method (using noindex)
• View All Method
• Rel Prev/Next Method

Each of these is detailed below.

Classic Pagination for SEO: Using noindex

I’ve already detailed this technique in full, so I’ll skip the nitty gritty. The important thing to realize is that using this method does not directly transfer any equity from a series of component pages to the primary, canonical page. Rather, as component pages get crawled and link back to the canonical page, that equity is (hopefully) transferred as a second-order effect.

We would generally not recommend using this method for pagination today, except for fringe cases. It’s perfectly fine and will not hurt a site; on the contrary, it will greatly help a site that has SEO pagination problems. But, there are now even better methods as we’ll discover.

The classic SEO pagination method uses noindex but does not directly consolidate equity.

View All Method

The most elegant method is to utilize a View All page. In this approach, all component pages rel canonical back to the View All.

There are a few requirements for this approach:

• The View All must load quickly; at least 3 seconds end-to-end. Maile Ohye pointed out at SMX West that even if load times are excessive, if the page can load progressively the user experience will not suffer as much (since content will be viewable on the page immediately).

At SMX West, a few folks complained when I mentioned 3.5 seconds as the maximum load time tolerable for View All pages. The truth is, this is a “real world” goal and while not ideal, reflects the actual load times that we see on large sites.

Just take a look at these ‘last mile’ load times on US retail sites to get an idea of what latency looks like out there. It’s not particularly pretty, but more than anything demonstrates the opportunity these sites have.

• 

  Site latency reports from Google Webmaster Tools

  Our analysis of 20 top ecommerce clients showed an average load time of just over 4 seconds. The fastest site was averaging 2 second load times, an exceptional result in this set. But it was more common to see load times above 3 seconds and well into the 4 second range. While the average load time was 4.2 seconds, the slowest site loaded in over 9 seconds!

Another requirement for the View All method is to ensure all products, or items, that are included on the component pages are featured on the View All itself.

This ensures that there won’t be anything left out of the crawl, as pages annotated with rel canonical tags will not necessarily have links within their HTML crawled. It will also ensure there is a relevant match between what is being folded together in the paginated series.

View All method elegantly passes link equity to the canonical

The benefits of this approach are two-fold:

1. Users tend to love view all pages. In our experience and testing, pages with a lot of products or items all featured at once convert much higher than landing pages with a smaller selection of products. But the pages need to be fast.
2. All component pages in the series transfer their equity to the View All in a fairly direct fashion.

Also something to be aware of: Google will attempt to use your View All page by default, all things considered, when there are no other proactive signals in place. Be aware of this and take steps to control the SEO experience proactively yourself.

The Rel Next/Prev Method

The most current technique for SEO pagination makes use of the HTML 4/5 link element rel=”next” and rel=”prev”. The specifics of this implementation are well detailed in this Google support page, so let’s focus on the benefits and results.

It’s been our experience (especially with e-commerce clients) that it can be difficult to get a View All implemented as the canonical and default page. Merchandising teams don’t always like them; they don’t make holiday or seasonal specials as easy to manage; advanced landing pages can be better looking and UX and content teams often prefer them; they can make spotlighting certain products more difficult; and many other reasons.

Because of these challenges, rel next/prev is often an excellent method for handling pagination.

The benefits of this approach are as follows:

1. All component pages share their equity with the series. What does this mean? Basically, when page 9 of a series gets a link with rich anchor text, that equity is shared across the series with all the other pages. That’s a good thing.
2. However, using rel next/prev doesn’t prevent a component page from displaying in search results. So while these pages will “roll up” to the canonical (or default) page 1, they could still fire at search time if the query was relevant for that specific page. At SMX West, Maile assured us that it would be a very rare thing for that situation to occur. But it could occur.
3. Because of this, an additional recommendation (strictly as an optional step) is to add a robots noindex, follow to the rel prev/next component pages. This would ensure that component pages would never fire at search time.
4. Finally, all rel next/prev pages should also have a self-referencing rel canonical tag. In cases where tracking IDs are appended to a URL, these rel canonical tags will ensure no duplication and equity leak occurs.

Ensure implementation of rel next/prev is thorough

Conclusion & Pagination Recommendations

SEO pagination needs to be recommended situationally (like so much of SEO). Here are my recommendations:

1. If you have a fast loading View All page, and that page contains all the products and/or items included across the component pages, use this method. All component pages rel canonical to the View All, and it becomes your default ranking page in SERPs. It’s elegant, simple, and efficient. It will also best pass equity from each page to a single, canonical URL.




2. If you don’t have a quality View All, or your company doesn’t want to use that as the canonical URL, implement the rel next/prev methodology instead. This method will consolidate signals across the series, rather than concentrate them on a single URL; however, the end result should be the same, if implemented well: the canonical, ranking URL (normally page 1) will be given the equity. There is a substantial benefit in using this method over the classic noindex approach: equity is actually transferred to the series itself.
   
   Remember, the classic method does not directly pass any equity – there are no signals to do so – rather it achieves the same ends by opening up the crawl of component pages and keeping them out of the index and from competing with the ranking URL. Be aware that with rel next/prev, component pages can still fire at search time (although unlikely). You can optionally use a noindex, follow as well to avoid this. Ensure all pages have self-referencing rel canonical tags.




3. There are edge cases where the classic noindex method of SEO pagination is still viable. These are in situations, for example, where it’s important to address Bing consistently along with Google (Bing does not yet support rel next/prev), or when HTML 4/5 elements are not yet ready to be deployed at an organization. In cases like these, the classic noindex method is still a good option.

No doubt this will change again, but here’s the latest for your SEO campaigns. Best of luck and please let me know in the comments your experiences and insights.

Updates: Google’s Maile Ohye has recently published a video on pagination and SEO. Be sure to check it out. Vanessa Fox also covers the details in her thorough treatment of the topic, Implementing Pagination Attributes Correctly for Google.

Pierre Far from Google posted on his Google+ that Google is slowing down GoogleBot’s crawl activity to reduce the effect on their site’s search rankings, if they did not follow the Google SEO advice from yesterday.

Pierre Far said:

Hello webmasters! We realize many webmasters are concerned about the medium-term effects of today’s blackout. As a precaution, the crawl team at Google has configured Googlebot to crawl at a much lower rate for today only so that the Google results of websites participating in the blackout are less likely to be affected.

Related Articles

• What All Marketers Need To Know About SOPA – The Stop Online Piracy Act
• #BlackoutSOPA: A Look At The Social Media Movement That Helped Stall The SOPA Legislation
• How To Blackout Your Site (For SOPA/PIPA) Without Hurting SEO
• Google Blackens Its Logo To Protest SOPA/PIPA, While Bing & Yahoo Carry On As Usual
• Why The Web Is Going Dark Over SOPA & PIPA

You may be thinking about joining the website blackout movement, but yikes … what about the SEO implications? How do you take your site offline in protest without messing up your visibility in Google’s search results?

Well, Google’s Pierre Far shared several tips earlier today on Google+ in a post called “Website outages and blackouts the right way.”

In short, the advice is to use a 503 HTTP status code to tell spiders that the website is temporary unavilable. With a 503 status, Google won’t index the content (or lack thereof if you’re blacking out your site) and it won’t consider the site as having duplicate content issues (when all of the pages are blacked out).

But Far adds a couple important caveats to this advice regarding the robots.txt file and what will happen in Webmaster Tools if Google finds your site blacked out. Another Googler, John Mueller, adds additional information in the comments, so you’ll want to read the original Google+ post if you’re thinking about blacking out your website this week for SOPA, or in the future for any other reason.

Of course, also keep in mind that Bing may not handle things the same way if you do blackout your site.

This gives Google’s spider, GoogleBot, the ability to read comments in AJAX or JavaScript, such as Facebook comments or Disqus comments and others that are dynamically loaded via AJAX or JavaScript. In addition, this means, Google is better at seeing the content behind more of your JavaScript or AJAX.

Postscript: Google now has an official blog post up with more details.

Related Stories:

• Google Offers A Proposal To Make AJAX Crawlable
• Google Offers SEO Advice On AJAX Coding
• Google May Be Crawling AJAX Now – How To Best Take Advantage Of It
• It’s Official: Google’s Proposal For Crawling AJAX URLs is Live
• An Update On Javascript Menus And SEO
• Google Now Crawling And Indexing Flash Content

Both James and I sent this information to Google as soon as we heard of it. After several hours, Google has told us, “we’re still investigating this report, and to be cautious we disabled all URL removals earlier this morning.” So now, if you even own a site, you won’t be able to remove the site or pages from the site using Google’s URL removal tool.

How did this loophole work? Pretty simple as James described. You use the following URL when logged into Google Webmaster Tools:

https://www.google.com/webmasters/tools/removals-request?hl=en&siteUrl=http://{YOUR_URL}/&urlt={URL_TO_BLOCK}

Then replace {YOUR_URL} with a URL you control within Webmaster Tools, and replace {URL_TO_BLOCK} with the URL of the site you want to block.

You could block a whole site, section or single page this way, based on how you entered the URL. To block a site, use the top level domain (E.g. http://www.someurl.com/), to block a section (subfolder) use a subfolder URL (E.g. http://www.someurl.com/somefolder/) and to block a page use the specific page URL (E.g. http://www.someurl.com/somefolder/somepage.html).

I am waiting an update from Google on why this happened, if site’s were impacted and how long this was an issue.

Postscript:: Google sent us a statement that they have fixed the issue. A Google spokesperson said:

We’ve confirmed that there was an issue within the URL removal feature in our Webmaster Tools and have already pushed out a fix and re-enabled URL removals.

The URL removal feature keeps detailed records, so we’re currently reprocessing earlier removal requests to ensure their validity. Our initial examination has shown only a limited impact.

Google no longer requires you to block access to the URL you want to remove from Google prior to submitted the URL removal request. Google said they have eliminated “the requirement that the webpage’s URL must first be blocked by a site owner before the page can be removed from Google’s search results.” Why did Google drop this requirement? Simply because since you have already validated and verified you are the owner of the site, Google felt it was redundant to require you to block the URL to prove again you are the site owner. Google explained:

You’ve already verified ownership of the site, we can eliminate this requirement to make it easier for you, as the site owner, to remove unwanted pages (e.g. pages accidentally made public) from Google’s search results.

Please note, these URL removals are only temporary and last 90-days. If you want the URLs and pages to never show in Google, you need to permanently remove them by 404ing the page or blocking them via robots.txt file or a noindex meta tag.

Related Stories:

• Removing Pages From Google: A Comprehensive Guide For Content Owners
• Google Releases Improved Content Removal Tools
• Google Lets You Tell Them Which URL Parameters To Ignore
• Up Close & Personal With Robots.txt
• Removing Your Personal Information From Google

This morning Rishi Lakhani tweeted about Google Translate pages appearing in Google search results. As you can see in the example below, pages with individual translation requests have been indexed.

All of the URLs that include a parameter seem to be individual translations. For instance, http://translate.google.com/?q=ART# displays as follows:

The problems with these types of pages being indexed in search engines is twofold:

• The Google webmaster guidelines say that Google doesn’t want to show search results in its search results and recommends that content owners block search results on their site from being indexed using robots.txt or a meta robots tag.
• That same guideline recommends blocking autogenerated pages from being indexed and a Google Webmaster Central blog a few months ago provided recommendations for handling machine-translated text so that it didn’t appear in search results.

A site owner might also want to block these types of pages from being crawled and indexed to increase crawl efficiency and ensure the most valuable pages on the site are being crawled and indexed instead.

I asked Google about this and they confirmed that indeed it was simply a matter of the Google Translate team not being aware of the issue and said they would resolve it.

Blocking Autogenerated Search Pages From Being Indexed

In the case of Google Translate, the ideal scenario is that the main page and any secondary pages (such as this tools page) be indexed, but that any pages from translation requests not be indexed.

Using robots.txt

The best way to do this would be to add a disallow line in the robots.txt file for the site that blocks indexing based on a pattern match of the URL query parameter. For instance:

Disallow: /*q=

This pattern would prevent search engines from indexing any URLs containing q=. (The * before the q= means that the q= can appear anywhere in the URL.)

In the case of translate.google.com (and all related TLDs), the robots.txt file that exists for the subdomains seems to be copied from www.google.com. Remember that search engines obey the robots.txt file for each subomain separately. Using the same robots.txt file for a subdomain that’s used for the www variation of the domain could have unintended consequences because the subomain likely has an entirely different folder and URL structure. (You can always check the behavior of your robots.txt file using Google Webmaster Tools.)

Adding the disallow pattern shown above to the www.google.com/robots.txt file would not work as search engines wouldn’t check that file when crawling the translate subdomain and in would instead cause search engines not to index URLs that match the pattern on www.google.com.

translate.google.com (and all google.com subdomains should have their own robots.txt file that’s customized for that subdomain.

Using the meta robots tag

If Google isn’t able to create a separate robots.txt file for the translate subdomain, they should first remove the file that’s there (and from other subdomains as well, as it could be causing unexpected indexing results for those subdomains). Then, they should use the meta robots tag on the individual pages they want blocked. Since the pages in question are dynamically generated, the way to do this would be to add logic to the code that generates these pages that writes the robots meta tag to the page as its created. This tag belongs in the <head> section of the page and looks as follows:

<meta="robots" content="noindex">

The technical documents are broken down into five section:

• Getting Started
• Robots.txt specification
• Robots meta tag and X-Robots-Tag specification
• Google’s crawlers
• References

I personally printed them out as my weekend reading.

# August 12th, MailOnline are looking for a talented SEO Manager so if you found this then you’re the kind of techie we need!
# Send your CV to holly dot ward at mailonline dot co dot uk

How clever! They suspect some of the best SEOs out there would be sniffing around their robots.txt file and used it to recruit a new SEO manger. If anything, it is getting the word out there via the press that they are looking for a new SEO.

This reminds me of when Google added the User-agent: zombies Disallow: /brains to their robots.txt file on Halloween. Also, in 2006, Brett Tabke used his robots.txt file as his blog for a period of time.

I do not believe anyone every used it to hire an SEO before.

That’s one of the big questions people are asking after yesterday’s launch of Facebook Questions. While many have assumed the answer would be “yes,” a Facebook spokesperson tells us that assumption is wrong.

Currently, search engines cannot access questions and answers through our Questions product. That may be something we consider for the future but have no current plans to allow it.

Facebook is blocking search engines by only showing Questions to logged-in users. Sure enough, a site:facebook.com/questions/ search on Google shows only a handful of results, none of which are actually Q&A from the first 24 hours that the beta has been open.

The same search produces zero results on both Yahoo and Bing.

Facebook’s move is unusual. As far back as 2007, the company was starting to open up some content to search engines — a trend that’s continued more recently with various agreements to let search engines access certain user content. For years, Yahoo Answers has been a pretty formidable rankings powerhouse, and no doubt gets a substantial amount of traffic via search engines.

It’s odd that Facebook would want to ignore that traffic source entirely. Don’t be surprised if this policy changes.

Postscript, July 30: Experian Hitwise has responded to my claim near the end of this article about how much traffic Yahoo Answers gets via search engines. In a tweet this morning, Hitwise reports that “62% of upstream visits to Yahoo Answers came via Google last week.”