The Leaky New York Times Paywall & How “Google Limits” Led To “Search Engine Limits”

How did we go from the New York Times seeming to single out visitors from Google with a “5 free visit per day” policy to supposedly all search engines being limited in such a way? And what’s up with there being no limit for visitors coming from anything that’s not a search engine, including Twitter […]

Chat with SearchBot

Nyt PaywallHow did we go from the New York Times seeming to single out visitors from Google with a “5 free visit per day” policy to supposedly all search engines being limited in such a way? And what’s up with there being no limit for visitors coming from anything that’s not a search engine, including Twitter and Facebook? Sit back, and I’ll tell the tale.

New York Times Paywall, Mark I

First, some history. The first New York Times paywall, TimesSelect, launched on September 19, 2005 and cost $49.99 per year (if you didn’t already have home delivery). It gave you access to New York Times columnists, archived articles and some premium content.

Paywall Dropped

TimesSelect lasted exactly two years, closing on September 19, 2007. Among the issues, as I recall them, was that by cutting off its content, the NYT also cut itself off from advertising pageviews. People couldn’t find stories through search; they couldn’t share them, either.Officially, the NYT wrote that user behavior had somehow radically changed over the two years of its experiment. I’ve bolded the key part:

Since we launched TimesSelect in 2005, the online landscape has altered significantly. Readers increasingly find news through search, as well as through social networks, blogs and other online sources.

In light of this shift, we believe offering unfettered access to New York Times reporting and analysis best serves the interest of our readers, our brand and the long-term vitality of our journalism. We encourage everyone to read our news and opinion – as well as share it, link to it and comment on it.

They Didn’t Know Search & Social Were Important?

Reading through this history now, all I can think is, “Are you f—ing kidding me?” Sorry for the profanity, even dashed out — but that was my reaction. Somehow from 2005 to 2007, people just suddenly started finding news through search. There was some major transition that happened then, that they didn’t forecast?

Please.

No, nothing like that happened at all. Trust me, I’d have remembered, if there was suddenly a new major search behavior change like that. There wasn’t. Certainly more newspapers realized in that period — as they have continued to realize — that search could send them traffic. But people were always searching on search engines for news.

News

Indeed, the reason the New York Times saw an explosion of new traffic was that during the same period it also began actively trying to attract search traffic seriously for the first time, as the NYT’s chief search strategist Marshall Simmonds said in 2007, in a good SEOmoz interview:

As many people have correctly pointed out, TimesSelect was certainly not productive from a link building standpoint. But in fairness, it can be difficult to communicate what the long-term value of being part of the conversation and getting these links will ultimately be.

Our SEO efforts at NYTimes.com had barely begun when TimesSelect was launched. Now, we have 2 full years of data that can reasonably predict the upside of our search optimization strategy.

High Quality News Costs!

Just over a year ago — in January 2010 — the New York Times decided it needed to try the paywall plan again. It wanted a system that would generate new revenue. How much revenue wasn’t said in this New York Times story about the announced plans, at the time. And people, we were told, needed to learn that quality news costs:

“I think we should have done it years ago,” said David Firestone, a deputy national news editor. “As painful as it will be at the beginning, we have to get rid of the notion that high-quality news comes free.”

And Readers Already Pay, Even If It’s Not Cash

Allow me a moment of mythbusting, here. The New York Times doesn’t currently give away high-quality news for free. When I show up there, I’m shown a bunch of ads:

Nyt Ads

If I actually buy a printed version of the paper, the subscription price I pay doesn’t come close to covering the actual production cost of the newspaper. Again, there are a bunch of ads that fund the paper.

I — and any reader of the New York Times — pays for much of what we read with our attention.

If our attention were worth nothing, there would be no New York Times, in print or on the web. Our attention is indeed valuable, which is why the New York Times has enticed us with its high quality news along with a mixture of other things. They want us to see those ads.

More and more of us, of course, no longer wish to have a hard copy of the paper delivered. So generally, it’s harder to make money off those hard-copy editions as print circulation drops.

Meanwhile, despite online “circulation” rising, the geniuses at the New York Times (not to mention virtually any other major newspaper you care to name) haven’t figured out a way in the 15 years or so we’ve had of the web to convince advertisers that online readers are just as valuable as offline readers, if not more so.

Begin weeping for them at any time.

Wherein Our Author Goes On An Entirely Skippable Tangent

Now another side note here, before I get further into the dissection of the woeful state of the NYT’s new paywall plans. I come from the newspaper world. That’s what I did, long ago, before I wrote about search.

I worked as an editorial researcher for the Los Angeles Times. I jumped ship to be a graphics reporter for the Orange County Register. I cut my teeth on reporting — and I loved newspapers and working for newspapers, not to mention just reading them. I still read two of them, hard copies even, delivered to my home.

But at the end of 1994, I remember seeing the web and knowing that it would be the future. I left the Orange County Register to do something with the web, because that paper’s debate over whether to do Prodigy, CompuServe or AOL for its online efforts convinced me I was going to miss out, that the newspaper industry itself was likely not to figure out that publishing was heading to the “platform” of the web.

Since that time, I’ve created and run two separate news web sites about search, each with paywall/membership offerings as well as free content. Both have been successful. Both have provided high quality news content.

It just happened that search was what I ended up covering and building sites around. But anyone with a passion, some smarts and free of the geriatric thinking that almost seems to be mandatory with the newspaper would could have done the same for a variety of other news topics. And goodness knows, the New York Times could have out Huffington Posted the Huffington Post, rather than whining about it.

Sorry for the detour, but perhaps all that is fair disclosure, a prism that I view these publications that cry about how they get millions of people reading them as a bad thing, even when they don’t always have the best content.

Don’t get me wrong. I love much of what the New York Times and other publications do. But they also get huge advantages based on their reputations, which makes it harder for the “new kids” to compete.

Planning The Paywall, Mark II

Anyway, back to the paywall. When it was first announced, we were told (again, bolding the key parts, as I will with all quotes going forward):

All visitors to NYTimes.com will have full access to the home page. In addition, readers will be able to read individual articles through search sites like Google, Yahoo and Bing without charge. After that first article, though, clicking on subsequent ones will count toward the monthly limit.

We were also told this had been carefully researched and plotted:

The Times Company has been studying the matter for almost a year, searching for common ground between pro- and anti-pay camps. Company executives said the changes would wait another year primarily because they need to build pay-system software that works seamlessly with NYTimes.com and the print subscriber database….

Most readers who go to the Times site, as with other news sites, are incidental visitors, arriving no more than once in a while through searches and links, and many of them would be unaffected by the new system. A much smaller number of committed readers account for the bulk of the site visits and page views, and the essential question is how many of them will pay.

The Times Company looked at several approaches, including a straightforward pay wall similar to The Journal’s, which makes some articles available to any visitor, and others accessible only to paying readers. It also rejected the ideas of varying the price depending on how much a consumer uses the site, and a “membership” format similar to the one used in public broadcasting.

The approach the company took was “the one that after much research and study we determined has the most upside” in both subscriptions and advertising, said Martin A. Nisenholtz, senior vice president for digital operations.

Remember, I’m quoting the high quality news content of the New York Times itself in explaining what it was doing.

Secret Tape #1: Inside The Planning Meeting

Now let’s imagine how one of the planning meetings went:

Executive 1: We need a new way to get money out of these freeloaders. What should we do, now that TimesSelect tanked?

Executive 2: Let’s block everyone!

Executive 1: No, dammit. We lose all that traffic. Didn’t you learn anything last time? We need something that gives the illusion of a paywall to idiots but lets all those pesky bloggers in.

Executive 2: Screw the bloggers. That’s all dying anyway.

Executive 1: No, one of them hits a paywall barrier, and they go off on a rant with lots of screenshots on how they can’t share stuff. Then the echo chamber fires up, and we look stupid. I’m tired of looking stupid.

Executive 2: Don’t we have some of those social media guru types that know about this stuff?

Executive 1: Yeah, let me get one. [Picks up telephone. Pushes button for assistant. Mumbles to get some social person up to the executive level]

Social Media Guru: Hey, this is easy. Just let anyone in who shares a link on Twitter and Facebook. Then we get all that sharing love, and no one hates us.

Executive 2: Is that easy to do?

Social Media Guru: Well, we can set up a “whitelist” of sites that we allow in. Of course, there are a lot of sharing sites. And then bloggers might want to link to us. If people click from a blog, and they get a paywall, that might generate hit. And there’s no way we’ll know all the blogs out there. Might be easier to just let anyone linking to us in for free.

Executive 2: Twit what? Is this easy to do?

Social Media Guru: Sure.

Executive 2: What about Google.  I hate those bastards. Can we block them?

Executive 1: Hold on there. Last time we got screwed by blocking them. Lost all that traffic and and stuff. I remember some SOO person or whatever going on and on about it.

Social Media Guru: You mean SEO.

Executive 2: Whatever.

Executive 1: Let me them in here. [Picks up telephone, gets assistant to get that SEO person to come up]

SEO Guru: Hey, good news. Rupert has..

Executive 1: Don’t use that name in here!

SEO Guru: Sorry. The people over at that big publishing corporation, has the name “News” in it, they’ve been yelling about Google ripping them off so much with First Click Free that they now allow it to be limited.

Executive 2: Huh? What?

SEO Guru: Well, a tiny number of people figured out they could read any of our articles if they searched for them and clicked from Google News. An even tinier number of people wrote these add-ons for their browsers that made it seem like they were coming from Google News. Those were pretty cool actually…

Executive 1: Shut up. What about the limits?

SEO Guru: Well, now you can limit things so that anyone coming from Google News can see only five pages per day. The beauty is, you know how we paginate the hell out of all our articles? Technically, they can only view the first five pages they come to from Google News. So heck, even if they come in on one of their five free clicks, we can screw them up when they go to the second page of a story.

Executive 1: This is great. Let’s implement that in our paywall plans. Does Google do this for us?

SEO Guru: No, Google just says it’s OK for us to figure out a way to do it.

Executive 1: Fine. You can go. Take that social media guru with you, too.

So In 2010, Search Visitors Limited After First View…

Now, this was all supposedly done and figured out before the announcement was made that the New York Times was going to try a paywall again, back in January 2010. How Google would be limited and so on, right?

Somewhat puzzling was that the New York Times report said that visitors could read articles through search engines but after the “first” article they read, reading more would count toward their limit.

That doesn’t make sense. With Google’s First Click Free, initially ALL articles you came to from Google had to be free. Then it was modified in December 2009 so that there could be a five visits per day limit.

This means the New York Times article back then should either have been saying that visits from Google would be unlimited or limited to five per day. Something wasn’t right in the whole “first visit free” thing that actually was announced.

In 2011, Only Google Gets “Five Visits Per Day” Limit

Fast forward to last week, when on March 17, the New York Times reported on how its new paywall would work — a paywall, understand, that took another year’s worth of planning and apparently, according to Bloomburg, $40 to $50 million to develop. From the New York Times story:

Beginning March 28, visitors to NYTimes.com will be able to read 20 articles a month without paying…

Not all visits to NYTimes.com will count toward the 20-article limit. In an effort to reduce losses among the Web site’s more than 30 million monthly readers, The Times will allow access to people who arrive at its Web site through search engines like Google and social networking sites like Facebook and Twitter. There will, however, be a five-article limit a day for people who visit the site from Google.

Why do we have these strange rules? They come out of that imaginary meeting above. Thanks to Social Media Guru, the social networking sites were given the all clear.

Thanks to SEO Guru, Google ended up with that five article a day limit. It wasn’t some figure the New York Times pulled out of the air. It was something based on the rules for Google’s First Click Free.

Unlimited Access From Any Link, Except Google Links

BioImage Martin A NisenholtzAlmost immediately, the loopholes started widening. Peter Kafka over at AllThingsD did an excellent interview with Martin Nisenholtz, senior vice president of digital operations for the New York Times (that’s Nisenholtz pictured over there on the right).

Peter Kafka: Just to be clear, when the Times says non-subscribers can read stories above their 20-per-month limit if they come from referring links, you’re not just talking about Twitter and Facebook, but any link from any site, right?

Martin Nisenholtz: That’s correct.

Kafka: It could be the Journal. It could be a blog, it could be the Financial Times, anything on the Web, right?

Nisenholtz: Yes. The only other thing is that Google has a methodology where they can limit the number of inbound links per day, and we intend to take them up on it.

Kafka: So that’s Google doing the actual gating, not you?

Nisenholtz: Right. They had made this feature available prior to us going pay, so it’s not like it was inspired by us per se. We’re just taking advantage of it.

I’m laughing now. Really, I am. First, to my understanding, Google doesn’t have a “methodology” where it does “gating” for publishers (and I have spent some time talking to them about this stuff). Google just tells publishers what the rules are, and the publishers implement those rules however they want.

Now, I’ll double-check on this later, and maybe I’ll be laughed at rather than doing the laughing. But right now, it sounds to me like the digital chief of the New York Times didn’t understand what SEO Guru was telling him.

More important, it’s not just people coming from social media sites that are allowed in for free. It’s people coming from ANY link. ANY LINK! Want to read a New York Times article? Tweet it to yourself and click on it. Blog it. Whatever. If it’s a link on a page, you’re in, no limits.

By the way, that’s also where Nisenholtz just makes no sense, when it comes to suggesting there’s a limit on Google in general. Google’s First Click Free is only for search results, not just for Google.

What happens if you subscribe to the New York Times by RSS feeds through Google Reader? Those are all links, so you should be unlimited. The same is true if you use Google Buzz, or click on a link to the New York Times from Google’s Gmail or come from a variety of other Google services.

Why’s Google Singled Out?

Back to Kafka, he’s not done. No sireee!

Kafka: Why limit Google’s links, but not any other sites?

Nisenholtz: I think the majority of people are honest and care about great journalism and the New York Times. When you look at the research that we’ve done, tons of people actually say, “Jeez, we’ve felt sort of guilty getting this for free all these years. We actually want to step up and pay, because we know we’re supporting a valuable institution.” At the same time we want to make sure that we’re not being gamed, to the extent that we can be.

OMG, I’m laughing again. “Why limit Google?” gets answered with “People are honest and care about great journalism?” Seriously? The honest answer would have been:

Um, I know SEO Guru told me we could limit Google, so I guess that’s why we do it. We can’t really limit all those other things like links from web sites and social sites and really any sites, if we want to have our paywall cake and eat the social media sharing frosting, too.

But What About The Binghole?

About PeterKafka keeps going [and giving me a serious mancrush as he keeps at it, so let’s get a picture of him this time, over there on the right]:

Kafka: But if you really do want to game the wall, you’ll be able to do it. You could could go through Microsoft’s Bing, for instance.

Nisenholtz: We’re obviously going to be vigilant over the next couple of months, in looking at the ways that people are doing that.

Here, Nisenholtz really steps in it. For one thing, I don’t even think Bing has a formal “First Click Free” style program like Google.

I’ll check on this, but the reason I’m uncertain is because no news publisher complains about Bing. It’s so little used compared to Google that it just doesn’t register as a threat, requiring Bing to roll out programs to appease the publishers, which in turn appear on my radar screen.

So in what “ways” will the New York Times block the Binghole? Especially since, according to what Nisenholtz said earlier, that type of gating is something that the search engines like Google do on their end.

Secret Tape #2: Dealing With The Google Problem

The answer, of course, is that the search engines don’t. This leads to another imaginary conversation:

New York Times Press Person: We’ve got a problem. Some reporter is asking about why we’re limiting Google but not Bing.

Executive 1: What’s Bing?

Executive 2: It’s that Microsoft search engine, I think.

Executive 1: Hmm, we’re blocking Google people after five visits but not Microsoft? That sounds kind of bad. Let me get that SEO Guru in here. [Calls assistant, you know the drill]

SEO Guru: What’s up?

Executive 1: Why aren’t we blocking Bing visitors after five visits?

SEO Guru: [Thinks to self: I suppose the real question is, why are we blocking Google visitors after five visits when we’re letting everyone else in from any link as much as they like]. We’re blocking Google visitors after five visits per day because they have a rule like that which allows us to limit in that way.

Executive 2: Can we do the same to Bing? Or other search engines?

SEO Guru: Maybe, if they have similar rules. Chances are, we can do whatever we want, and they probably wouldn’t even notice.

Executive 1: OK, you can go. [After SEO Guru leaves, to Press Person]: Tell that reporter that we’ll limit other search engines, too.

Now, All Search Engines Will Be Limited

Alexia1That brings us to the article from Alexis Tsotsis over at TechCrunch (that’s her, over there on the right), where the New York Times now says all major search engines will face a five visits per day limit:

After reviewing our options, we decided to extend the policy of five free clicks per day to all major search engines by the global launch on March 28. Our pre-launch period in Canada was undertaken to enable us to test the systems and fine-tune the model.

OMG, I’m laughing again. Really, I’m having a good chuckle over all of this. Where to begin?

Limited By Fairy Dust & Other Imaginary Plans

First, which “major search engines” are we talking about? Google, Yahoo and Bing? How about Baidu? Hey, YouTube is the second largest search engine in the US — what happens if someone links to you from a video there?

Facebook has search results that come from Bing, plus it has its own custom links that lead to pages — and it has enough search activity to be a considered a major search engines. Does Facebook face a limit?

And you figured all this out during your Canadian test. You’re just killing me now. Seriously, just killing me. You’ve had an entire year to plot this, and a year before that when you were also supposedly figuring it out, and it was only in the last week that you determined that Bing should get limited like Google?

And what, over the past year when you almost certainly were already limiting Google visits already, even when you had no paywall, the whole Bing issue didn’t come up?

Also, this is kosher with all those other unnamed major search engines. What, in the past few days they all magically settled on a uniform “five visits per day” figure just like Google has, for First Click Free programs they probably don’t even have?

This whole thing is a giant joke, and I haven’t even gotten into the issue of how easy it’ll be for anyone who actually does hit the paywall ind the end to get around it in various ways, from defeating the Javascript code to using plug-ins that’ll come. Joshua Benton over at Nieman Journalism Lab looks at this more; the opening paywall image at the top of this article comes from his story.

So Much Worry About Limits No One Will Hit

The bottom line is what the New York Times told Kafka elsewhere in that interview with him:

But in looking at the research that we did, we expect [paywall jumpers] to be a very significant minority, a small, small number of people.

Of course it will be a minority, because practically no one is going to hit this wall. That’s why this from what the New York Times reported about its own paywall is so disturbing:

No American news organization as large as The Times has tried to put its content behind a pay wall after allowing unrestricted access. The move is being closely watched by anxious publishers, which have warily embraced the Web and struggled with how to turn online journalism into a profitable business.

It’s just not accurate. So much of the New York Times site is open for so many people that this really isn’t a test of restricting much of anything.

The Few, The Blocked, The Loyal Stupid & Grateful

Subscribe Nyt

This wall is designed, as best I can tell, only to be a barrier to your most loyal — and most stupid — readers. People who come to you each day, because they depend on you as their starting point and don’t realize that you’re providing all this content to drive-by visitors for free. They cough up, because they’ll assume they have to.

It’s also designed to earn off your most loyal readers who aren’t stupid and do value you. If I depended on the New York Times each day, really depended on it, you bet I’d pay. And gladly, too. And despite the idiocy of how your so-called “paywall” is designed, I hope you do earn off both the stupid loyal and the grateful loyal.

But What About The New Digital Generation?

There’s plenty of dissection going on about your paywall — Mediagazer is full of stories and analysis on the topic. But all the loopholes and ways to get around the wall is beside the point, to me.

What worries me more is what you’re not doing — figuring out a way to convince all those new digital readers to also pay. Let me quote from what one of your reporters David Carr wrote two days ago:

Plans like this usually get met with derision at events like SXSW. It is an article of faith among the digitally inclined that only losers pay for – or try to charge for – content. After all, the Web was built on collaboration, open networks and a friction-free flow of information. And The Times’s attempt – however considered, however nuanced – is an offense against that theology.

I’ve met David. I like David. But I disagree. I know many “digitally inclined” people who pay for all types of content. But they pay for things of value. You’re offering little of value to them, with this new plan, except maybe those who’d like access via apps.

You’re not having the guts to seriously inconvenience them with a real paywall — where some people actually would pay to be rid of that.

You’re not offering them a simple, no brainer choice such as John Gruber writes about, say $25 per year to clear a nag overlay that you might put on top of all your pages. Don’t pay the fee, keep clicking to clear the damn box. Pay it, get rewarded with convenience. And damn those who complain.

You’re not offering them a range of other benefits that you should have been figuring out over the past year, or two, or fifteen years, to really secure your future. That’s your future audience, and all you’ve really trained them to expect is that there’s no need to pay.

(featured home page image via Shutterstock.com)

Postscript: I contacted the New York Times with these questions:

  • Why have a limit on search engines at all, given you’re allowing links from anywhere else to let people in?
  • Does Google really “gate” content for you, or is that actually done by the NYT?
  • Is Bing allowing you to have paywalled content in Bing and if so, how and through what program.

So far, I have only received this statement from New York Times communications manager Kristin Mason:

Thanks for reaching out. At this time, we have nothing more to offer beyond the statement we have provided: After reviewing our options, we decided to extend the policy of five free clicks per day to all major search engines by the global launch on March 28. Our pre-launch period in Canada was undertaken to enable us to test the systems and fine-tune the model.

Postscript 2: Now I’ve heard back from Bing. It has confirmed for me that the search engine has no formal support for subscription content, nothing like Google’s First Click Free program, though it might add support for this in the future.

On occasion, Bing will work one-on-one with a publisher to help with paywall and registration wall issues. But Bing didn’t say it was doing anything special for the New York Times. Said spokesperson Scott Massey:

I don’t believe they’ve shared any details, and we aren’t able to comment on the their subscription plans.

Nor would the New York Times need to have any special help, because Massey also explained that since Bing lacks any formal program, publishers are free to do what they want:

Bing leaves it up to the web site/publisher. They can either allow the BingBot crawler past the wall [paywall or regwall] or not. If they let the crawler past the wall, then they can determine how much or how little of that content someone has access to.

In short, if the New York Times can concoct a way on its end to block Bing visitors after 1 visit, 5 visits or 500 visits, Bing doesn’t care. It’s the New York Times that’s doing the gating, not Bing.

Postscript 3: Heard back from Google now. It has confirmed that it does NOT do any “gating” for the New York Times or any other publisher and that if a publisher wants to limit according to First Click Free’s 5 visits per day allowance, that’s up to the publisher to implement.

Postscript 4: See my follow-up post now that the paywall is live, The New York Times Paywall Meters All Google Visits, Not Just Search Visits

MVW: too many? Pam do one?

Or tools, like Mechanical Turk, that allow you to outsource SEO drudgework for astonishingly little cost?

Q&A Sites for Google Survival?

Being seen in the newsfeed,
getting shared, comments

(include sculpting?)


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.


About the author

Danny Sullivan
Contributor
Danny Sullivan was a journalist and analyst who covered the digital and search marketing space from 1996 through 2017. He was also a cofounder of Third Door Media, which publishes Search Engine Land and MarTech, and produces the SMX: Search Marketing Expo and MarTech events. He retired from journalism and Third Door Media in June 2017. You can learn more about him on his personal site & blog He can also be found on Facebook and Twitter.

Get the must-read newsletter for search marketers.