SEO

The New York Times Paywall Meters All Google Visits, Not Just Search Visits

The New York Times launched its paywall globally today, and I’ve been putting the barrier through its paces. Visits from search engines like Google, for convoluted reasons, are limited to five per day. But as it turns out, visits from any Google site — search or not — appear blocked. To test how the new […]

Danny Sullivan on March 28, 2011 at 9:49 pm

The New York Times launched its paywall globally today, and I’ve been putting the barrier through its paces. Visits from search engines like Google, for convoluted reasons, are limited to five per day. But as it turns out, visits from any Google site — search or not — appear blocked.

To test how the new paywall works — and interacts with search engines — I triggered it for myself in Firefox. At first, I had no luck. After clicking on well past the allowed 20 views per month, I couldn’t get the wall to appear.

Restarting my browser (I was in Firefox) finally got the paywall to trigger. As was promised, I was still able to get past this when I came from links on social sites such as Twitter. In addition, I was able to get past the paywall when I came from an ordinary link, such as from River Of Times, which is a collection of New York Times links.

Blocked From Google, Yahoo & Bing

Search engines are also supposed to let you past the paywall, if you’ve hit it, but only for up to five visits per day. That’s a barrier that the New York Times has erected against whatever it considers to be a “major search engine.”

To discover which search engines got this special attention, I went to the Google.com, Yahoo.com, Bing.com, Ask.com and Blekko.com, found five stories from the New York Times in each of them, and clicked through.

When I tried to view more than five, I was blocked when coming from these:

Google
Yahoo
Bing

I wasn’t blocked when coming from Ask and Blekko. In addition, I wasn’t blocked when I came from any of these country-specific versions of Google that I tested:

Google Australia
Google France
Google UK

With Google UK (google.co.uk), I specifically tested to see if I could view more than five articles, in case the New York Time was perhaps applying a different limit for each version of Google. This doesn’t appear to be the case. I was able to view more than five pages, when coming from Google UK.

Blocking Google Reader

Next, I wondered if the New York Times was only metering Google search-related visits (which is what the New York Times said it would do) or instead metering visits from other Google services. As it turns out, any Google service may get blocked.

For example, I subscribe to the RSS feed from the New York Times in Google Reader. Once the paywall was triggered, I could no longer view articles from Google Reader without a paywall block coming up.

This shouldn’t happen. For one, the New York Times has said that links from social media sites wouldn’t be blocked, and I’d say Google Reader qualifies as a social media site. But more important, the New York Times has also previously said that ANY link from another site will let visitors through, unmetered, except in the case of search engines.

Google Reader isn’t a search engine, but the New York Times is treating it that way. More crazy, if I subscribed to the RSS feed in my browser, the New York Times consistently let me view whatever I wanted.

Blocking Google Buzz

Few would argue that Google Buzz isn’t a social media outlet. It might not be anywhere near as popular as Facebook or Twitter, but it is clearly a social media service. But in my testing, I wasn’t able to view an article I shared on Google Buzz, because I’d hit the paywall limit.

Again, this shouldn’t have been the case.

Blocking Gmail

I tried another test. I emailed myself a link to a New York Times article to my Gmail account, which is run by Google. Again, I got blocked. The same thing happened when I clicked from an emailed link sent to my Yahoo Mail account.

I’m pretty sure that if I’d clicked on these links from an email account not also run by a major search engine that I’d have gotten through — but that really shouldn’t be the case.

Postscript: HTTPS To Blame?

After I wrote my story above, a reader wrote suggesting that perhaps the secure connection (https) that Google provides for services like Google Reader, Buzz and Gmail might be to blame. The connection might not be passing “referrer” information that indicates someone is coming from these services, and so the paywall goes up.

I don’t think this is the case, but it’s worth noting. I did some further testing on the morning of March 29, the day after I wrote this. I found:

Clicks from Google Reader were now being allowed through, regardless of whether someone comes from a secured connection or not
Clicks from Gmail continue to be blocked, even when I turned the secure connection off temporarily
Clicks from Google Buzz continue to be blocked, even with the secure connection off

I also found clicks from Twitter, even from a secure connection, were allowed through the paywall. So, there’s more going on. Secure connections can pass referrer information, if they want, to my understanding. It’s also possible that the secure connection might redirect a click in a variety of ways to pass on different referrer information.

More Paywall Observations

The paywall is pretty leaky. It’s not going to stop anyone who wants to get past and knows even a little bit about the internet. Aside from admitting people via any link, you can also get through by:

Using the private or “incognito” mode of modern browsers like Firefox, Chrome and Internet Explorer
Clearing your nytimes.com cookies (and probably just one specific cookie, but I haven’t pinned that down)
Using another browser, if you have more than one on your compute

There are other tips, too, such as those Mashable has compiled.

You’re Blocking Search Engines Why?

The purpose of me writing about this isn’t to encourage people to avoid the paywall. Heck, the New York Times is deliberately going out of its way to keep it from triggering for many people who come to the site.

Instead, it’s to revisit the absurdity of how search engines ended up getting slapped with one of the few limits out there. It all started with the New York Times trying to keep limits that Google allows publishers as part of its “First Click Free” program, limits that make little sense when pretty much it’s “Any Click Free” now at the New York Times.

After questions were raised about why Google was getting singled out, the New York Times then had to concoct a story about how all major search engines would get blocked, with a confusing message suggesting that the New York Times was using tools that the search engines themselves provide for this.

They don’t. The blocking and metering is all done by the New York Times itself — and done so poorly, right now, that it is inadvertently blocking non-search services from these companies. Personally, I think the search engine limits should just get dropped.

Further Background

My article below covers more background on the New York Times blocking of search engines and the confusing statements it has made:

The Leaky New York Times Paywall & How “Google Limits” Led To “Search Engine Limits”

Over at my personal blog, I also did a send-up of the pricing plans for digital subscriptions (in short — buy a print subscription. You get digital access, should you really think you need it, for a much better price)

A Better Letter To New York Times Readers About Digital Subscriptions

And below are some further background articles on search engines and newspaper sites.

About The Author

Danny Sullivan

Danny Sullivan was a journalist and analyst who covered the digital and search marketing space from 1996 through 2017. He was also a cofounder of Third Door Media, which publishes Search Engine Land, Marketing Land, MarTech Today and produces the SMX: Search Marketing Expo and MarTech events. He retired from journalism and Third Door Media in June 2017. You can learn more about him on his personal site & blog He can also be found on Facebook and Twitter.