Google uses the location of a site in its ranking algorithms. A searcher in the UK is more likely to see sites from the UK in results. But for top level domains that aren’t restricted to a particular country, Google uses other signals, such as the location of the server in determining what country a site is most relevant for.

Site owners can specify the target country for these generic top level domains in Google Webmaster Tools (but can’t specify a different target if registration of the TLD is restricted to a specific country). Thanks to Googler Pierre Far for letting everyone know!

This is a great solution, but over time, more countries (such as Columbia, with their ccTLD of .co) have opened up registration. Site owners with these domains have been frustrated that Google hasn’t supported the new generic nature of the TLDs. A .co could end up ranking only in Columbia, even if the site didn’t target users in that country.

Now, Google has expanded the list of ccTLDs that they recognize as generic. So if you have a .co or .io, you can now specify the country that it should be associated with. (and @dotco is pretty happy about that.) As always, if your site is not country-specific, don’t specify a target country. Google’s index now recognizes that the unrestricted ccTLDs shouldn’t be associated specifically with those countries so they won’t be seen as more relevant for those users.

Great news for owners of these types of domains and for searchers.

Bing tells me that while they still support the #! version of crawlable AJAX originally launched by Google, they’re finding it’s not implemented correctly much of the time, and they strongly recommend pushState instead.

Why AJAX Can Be Difficult To Crawl & Index

One common use of AJAX is to make the website experience faster for a visitor, but this implementation can have drawbacks for SEO. Imagine, for instance, a page with several tabs of content.

A web developer could implement this one of several ways.

• A separate URL for each tab – with this implementation, when the visitor clicks a tab, a new request is made to the server for a completely new page. The URL structure might look something like:
  • http://www.example.com/my-web-page?tab=one
  • http://www.example.com/my-web-page?tab=two
  • http://www.example.com/my-web-page?tab=three
• CSS for each tab – with this implementation, the server returns the contents of all tabs with the first page request. When the visitor clicks a tab, the CSS rules cause the browser to hide the contents associated with one tab and show the contents associated with clicked tab. Only one URL is associated with the page, like this:
  • http://www.example.com/my-web-page
• AJAX rending of each tab – with this implementation, when the visitor clicks a tab, only the changing portion of the page is replaced. The URL structure might look something like:
  • http://www.example.com/my-web-page#tab=one
  • http://www.example.com/my-web-page#tab=two
  • http://www.example.com/my-web-page#tab=three

As with most things, pros and cons exist for each option. A separate URL for each tab is easy to share and bookmark and is easy for search engines to crawl and index (they can extract all of the content from each page and have a separate URL to associate with each), but reloading the entire contents of each page can be slow.

CSS for each tab is also easy for search engines to crawl and index, and in some cases, the combined page may rank higher than the same content broken into three pages (due to consolidated incoming links and relevancy signals). But the request for all of that content at once can be the slowest of all to render, and users can’t share or bookmark the page with a secondary tab as active.

AJAX rendering is fastest and enables easy sharing and bookmarking. But, search engines have historically had the hardest time with this implementation. Search engines have trouble extracting content from AJAX/JavaScript calls (although Google has been getting better at it). And # in a URL started as a way to link to content within a page, and so search engines tend to ignore everything in a URL past the #.

Crawlable AJAX

In 2009, Google put together a way to make AJAX crawlable. With this method, a the webpage would use #! rather than #, like this:

• http://www.example.com/my-web-page#!tab=one

For a normal user agent, such as a browser, the # would trigger the AJAX portion of the page, just as it would in a normal AJAX implementation. However, a search engine user agent such as Google would see the #! section of the URL and then request a special version of the page (replacing #! with ?_escaped_fragment_=). In response, the server would return a static version of the page with the contents normally rendered through JavaScript. The benefits of this implementation were that search engines could associate a separate URL with each set of content; and even better, could extract all of that content.

In 2011, Bing began supporting this implementation and included a checkbox in their webmaster tools so site owners could let them know it was being used on a site (they’ve since removed the checkbox, as they’ve gotten better at detecting and crawling it).

As with the other implementations, this has its drawbacks as well, not least of which is the complicated implementation. Bing’s latest blog post notes:

“Developers had to rely on overly-complicated protocols such as “crawlable AJAX”, which uses the #! (“Hash bang”) signature.  This was meant to make AJAX development easier in regards to SEO, but it actually complicated things both for search engines as well as webmasters trying to implement, maintain, and debug their AJAX-driven web pages and applications.

With pushState, we can fully omit the complexity of transforming between “pretty” AJAX URLs and “ugly” static URLs. Search Engines will crawl and index the same URL used by you customers. We are back to business as usual for SEO, including pretty SEO well-understood URLs schema http://domain/path/file?name=value_parameters. This helps you focus on the usual SEO activities (links, page content, etc.) without having to do worry about complex page transformations.”

I asked Bing’s Fabrice Canel, who wrote the post, if Bing still supports the #! version of AJAX URLs, and he told me:

 ”We are still supporting the #! crawlable AJAX method but as I said, we do not recommend it at all and we really prefer pushState which is far easier for webmasters and web developers to adopt and maintain.”

HTML5 pushState

With HTML5 pushState, pages can take advantage of the best of both worlds: URLs without # (so search engines can easily index them) and dynamically rendered content for only the change portion of the page (to make things as speedy as possible).

With pushState, URLs look like the first example (a separate URL for each tab), but operate like the third example (AJAX rendering of each tab and the resulting URLs look as follows:

• http://www.example.com/my-web-page?tab=one
• http://www.example.com/my-web-page?tab=two
• http://www.example.com/my-web-page?tab=three

There are other, more complex ways of getting to this same result, such as Hijax, but pushState can be much easier.

Google has been supportive of HTML5 since the beginning (Google’s Maile Ohye in particular began recommending it in conferences since early 2012) and recently published a video in support of HTML5 pushState.

Click here to view the embedded video.

In the video, Google’s Matt Cutts noted:

“A correctly implemented site that uses pushState typically doesn’t need any extra support in order for us to be able to crawl it. We do support both [pushState and #!] and we do handle both standards… but [pushState] is something that I would encourage you to look into and it can be quite helpful in making sure things can be crawlable.”

And now Bing has announced support as well.

Of course, HTML5 has its drawbacks as well, notably that not all older browsers support it, and can require significantly engineering resources to implement (as you’re replacing the current site’s HTML implementation).

If your site uses AJAX-based URLs (either the # versions noted above, or versions that don’t change at all when  content changes) and subsequently, the site is not fully crawled and indexed and you’re looking for solutions, HTML5 pushState is definitely worth looking into.

If your site uses the crawlable #! URLs and isn’t having any trouble being indexed, then you can leave things as they are for now. Both Google and Bing continue to support this implementation.

If you’re considering adding AJAX to your site, make sure you think through the implementation carefully and take into account what content search engines can extract and whether the URLs are indexable.

Related:

• Bing Now Supports Google’s Crawlable AJAX Standard
• Google Can Now Execute AJAX and JavaScript for Indexing
• How to Best Take Advantage of Google’s Crawlable AJAX

For some time, Google’s been letting site owners know when their sites have been hacked, but this new help center takes the next step and walks through how to fix the site and get the warnings removed from Google’s search results. Although as Maile notes in the blog post: “while we attempt to outline the necessary steps in recovery, each task remains fairly difficult for site owners unless they have advanced knowledge of system administrator commands and experience with source code”.

The blog post also provides steps for avoiding getting hacked in the first place: “Just as you focus on making a site that’s good for users and search ­engine friendly, keeping your site secure ­­ for you and your visitors ­­ is also paramount.”

The help content is a collection of videos (totally over an hour in length) and articles that provide comprehensive and detailed information about how sites get hacked and why, and spam techniques and how to detect them, in addition to explanations about how to recover from the hacking. It’s all pretty fascinating stuff, even if your site hasn’t been hacked (well, fascinating if you’re kind of geeky, like I am).

Click here to view the embedded video.

Related:

• Google: 12 To 14 Million Searches Per Day Returned Hacked Sites
• Google Sent 20,000+ Hacked Notification Messages To Webmasters Today

For the last five years, I’ve tracked where advertisers are trying to send viewers, where those viewers are actually going, and in particular, if advertisers are taking full advantage of the furious searching that happens post-game. Both Google Trends and Yahoo data show that even as we turned to Twitter in droves, we also, as in past years, flocked to the search engines.

Google Trends: Super Bowl Sunday 2013

Yahoo Search Data: Super Bowl Sunday 2013

Last year brought a record high of brands advertising their own websites. This year, that trend was down, replaced by hashtags. If the goal of a Super Bowl is to get people talking about you, how better to encourage that than to give a little nudge and a way for those talking to find each other. (You can find the full infographic with these images on my site, Nine By Blue.)

But, brand confusion crept in online. Many online versions of the commercials sent viewers to different places than the TV versions, and the paid search ads sent searchers somewhere different still. Several brands had one owned site ranking in unpaid results, two different sites showing up in paid search results (at the same time!), and a different site still advertised in the ad.

Many sites are better than none, as in the end, your potential customers are finding you… somewhere. But this structure is not an ideal way to built up long term search value (to take full advantage of the engagement and links that a Super Bowl commercial brings), and it seems impossible to measure success. Is no one going to the special microsite built just for the commercial because they didn’t find the commercial engaging? Or, is it because no one knows the microsite exists?

Search Visibility

This year, those searching for Super Bowl advertisers could mostly find them! The majority of brands bought paid search ads for their names, and half bought paid search ads for their advertised taglines. Every single advertiser other than Fast and Furious 6 had a website that ranked in the unpaid search results.

Brands were harder to find for those searching for taglines. Half of the advertisers did realize someone might search for those though, and bought Adwords, which is much better than past years.

Paid Search

Eighty percent of brands bought AdWords for branded searches and 50 percent bought AdWords for commercial taglines, but where were these ads sending people?

Mostly, the ads were sending people to an official site. This was great, although I’m giving some brands a little too much credit, as in some cases, these ads seemed to be part of long-running AdWords campaigns unrelated to the Super Bowl. This became more apparent when ads for both the official site and a microsite or YouTube URL showed up. The agency (or department) managing the Super Bowl commercial may not have been talking to the agency (or department) managing the regular AdWords spend. That can result in a company bidding against itself, paying more than it has to for ads, and confusing searchers.

How’s The Searcher Experience?

Let’s take a closer look at a few brands.

Dodge Ram Trucks

The Ram truck ad was popular among viewers and generated a lot of search volume. Google Trends showed that it was one of the most popular searches on Sunday, and Yahoo told me that it was the most searched for commercial.

But, after doing related searches, I ended up totally confused. Is a Ram a Dodge? Are Ram trucks and Dodge trucks different?

The paid search results for [ram trucks] show no less than four ads that could be the official site (and three that say they are). The first result — www.ramtrucks.com — is, in fact, the best place to send Super Bowl searchers.

It ties in perfectly with the commercial, lets you watch it again, and showcases the trucks. Dodge is advertising www.dodge.com against the [ram trucks] search, and its website doesn’t even show Ram trucks. If you didn’t know Dodge owned Ram, then you’d think it was a competitor, swooping in and trying to lure potential customers away.

Mercedes Benz

Mercedes may also have had uncoordinated campaigns going as a search for [Mercedes] showed a paid search ad for mercedesbenz.com, and a search for [Mercedes CLA] (the car advertised in the commercial) showed a paid search ad for the page within the mercedesbenz.com site created for the commercial.

I can’t be too hard on them, though. To their credit, the home page features the ad and links to the Super Bowl page, and the Super Bowl page itself is actually on the mercedesbenz.com domain and not on a separate microsite (that has to be maintained separately and has to start from scratch for search engine indexing and ranking). They’ve also got a Google+ page with recent posts, which makes their Google search results look even better for their brand.

Volkswagen

Volkswagen had great intentions, and in many ways, great follow through. They released the ad early and got people talking and they included a hashtag that helped build buzz on Twitter.

But then things get confusing.

A search for [vw] returns what are likely regular branded AdWords, unrelated to the Super Bowl commercial.

As with Mercedes, that’s not too much of a problem, since the home page features the commercial.

Searches for variations for the tagline (“get happy” and “get in get happy”) brings back an ad with a display URL of www.vw.com/gameday (which 404s if you type it in directly) that redirects to the home page. Totally fine, except that [get happy] doesn’t return VW in the unpaid results, and something peculiar happens in the unpaid results for [get in get happy]. A microsite shows up!

Look, I totally get the reasons brands have to use microsites. It may be utterly impossible to get the content onto the main site due to technical reasons or bureaucratic red tape. But why invest so much into a separate site and then not advertise it? Maybe because it’s broken. When I try to load the microsite with Chrome, I just get a guy in a Beetle, with a directive to click on his head. Doing so, however, does nothing. That doesn’t make me get happy.

The full site loads for me in Firebox, but apparently requires a Facebook login (although that’s not clear until you start trying to use it). I’m still not happy.

And the videos (both the commercial and a funnier video that shows YouTube famous sad people getting happy) play in an overlay with no way to share them. Now, I’m sad.

Huyndai & The Secret Microsites

Hyundai has an even more secret microsite, as they don’t seem to be advertising it anywhere anymore. When the #pickyourteam commercial aired online before the Super Bowl, it advertised a microsite: www.findyour7.com. But when it aired on TV, it showed a hashtag instead: #pickyourteam. It makes total sense to drive online viewers to a website and TV viewers to an active discussion, but why use two different phrases?

They also put together a microsite for another of their ads: www.epicplaydate.com, but they didn’t advertise this anywhere but in the paid search ad that appears for [hyundai santa fe] (even though both the #epicplaydate and #pickyourteam ads were for the Santa Fe). The paid ad for [epic playdate] searches is for hyundaiusa.com. As is the ad for [#pickyourteam] searches. (Searches for [pick your team] didn’t trigger an ad for me).

Confused yet?

Of course epicplaydate.com doesn’t rank in unpaid results for [epic play date] and findyour7.com doesn’t rank for well, [pick your team] because the microsites have similar problems as Hyundai’s microsite during the 2009 Super Bowl: they don’t actually have the words in the page that people are searching for.

In the case of epicplaydate.com, if you have JavaScript disabled, all you see is “loading.” Oddly, that’s not what Google sees, as the cache version of the page is full of everything about the Hyundai Santa Fe (other than the words “epic playdate”) and claims to be for allnewsantafe.com (which 302 redirects to epicplaydate.com) so some weirdness is going on that requires some additional digging.

www.findyour7.com does rank for [find your 7]. If only anyone knew to search for it.

The website doesn’t mention the Super Bowl commercial. Maybe they realized that the commercial only had six kids on the team. And he only had to pick five.

Most Popular Topic of the Day: Beyonce vs. The Blackout

According to the (sadly now significantly scaled back) Google Trends, the popular topic of of Super Bowl Sunday was Beyonce, with over one million searches.

Yahoo also found Beyonce to be popular, as related searches spiked  7236% (and searches for Destiny’s Child rose 4000%).

But according to Yahoo, the big winner of the day was the blackout, as viewers turned to their devices to ask [why did the lights go out], [superdome power outage] and [what caused power outage].

It may have taken just four minutes for someone to advertise on related terms of Twitter, but did anyone but the terms on Google? In order of popularly, Yahoo found we were most interested in:

• the blackout
• Beyonce
• the commercials
• kickoff
• watching online

Google’s blog post  today on Super Bowl search activity found M&Ms as the top trending search and found interest in the blackout was the eighth most popular topic. M&Ms? Really? Some trending searches during the game from Google, over time:

The Power of Commercials

As always, I’ll be doing more in depth analysis of the commercials (check back for that tomorrow), but which ones caused us to search the most yesterday? According to Google Trends, we were most interested in:

• Ram trucks (Paul Harvey: God Made a Farmer)
• GoDaddy
• Iron Man 3
• Fast and Furious 6

Yahoo also found that the Ram trucks commercial drove the most interest and found that in addition to what Google listed, we were mostly searching for:

• Taco Bell
• Doritos
• Budweiser
• Tide
• Oreo
• Calvin Klein

As noted earlier, Google found that the most searched for commercial was for M&Ms, although this was apparently on YouTube, which still seems a little… unusual.

Yahoo data also shows the power of a Super Bowl commercial for raising awareness of an unknown brand as searches for Sodastream spiked 5244%.

And although Hyundai sponsored the pregame show and had the most commercials of anyone, according to Yahoo, the most popular car brands during the game (after Ram) were Jeep, Kia, and Mercedes and According to Google they were Mercedes, Lincoln, and Audi.

The Popularity of Celebrities

Every year, I note that advertisers who use celebrities in their commercials should not only think about searches for their brand, but about searches for the celebrities as well. Commercials with celebrities always cause search spikes for those celebrities and often, the brand is nowhere to be found in search results. That trend continued this year, with Yahoo finding the top five most popular celebrities:

• Kate Upton
• Kaley Cuoco
• Stevie Wonder
• Paul Rudd
• Seth Rogen

Searches for Oprah (who voiced the Jeep commercial) jumped 7o%.

But What About The Game?

Apparently, a football game was played yesterday and some viewers were interested in that! Google found that the Ravens were searched slightly more often than the 49ers and that the most searched for players were:

• Colin Kaepernick
• Joe Flacco
• Michael Oher
• David Akers
• Jacoby Jones

This year, Google just provides the answer right at the top of the page: February 3rd at 3:30 pacific.

Just as we saw last year, the NFL understands what their target audience is looking for and has built a page that provides exactly what they need (that ranks first in the search results), as well as invites them to tour the rest of the site (which gives the NFL the page views they need – win/win!).

Super Bowl Start Time: Search Results Through the Years

This query is a great example of how far search has come (both for search engines and for organizations who want to reach their audiences online). Let’s take a quick journey back in time.

2009 Super Bowl - Google offers up the 2007 game (and Microsoft Live Search provides a Wikipedia result about the history of the Super Bowl).

2010 Super Bowl -Google attempts to fix the problem of choosing either spammy results or spammier results (“So when is the Super Bowl 44 start time? What is the Super Bowl Time? Football fans all over America would be anxious for the game to begin on 2010 Feb 07 at Land Shark Stadium, Miami, Florida. The Super Bowl 2010 start time is pretty well set…”) to rank by adding a Onebox with the start time at the top of the page. (Bing mostly just provides the spam.)

2011 Super Bowl – Google gives news organizations the benefit of the doubt, and puts Google News results above the Onebox that gives searchers the answer they are looking for.

Sadly, news organizations have decided that what works for churn and burn spam sites works for credible organizations with names that would hurt more to lose and thus began HuffPo’s tradition of crafting articles almost solely out of search queries (“Are you wondering, “what time does the Superbowl start?” It’s a common search query, as is “what time is the super bowl 2011″, “superbowl start time” and “superbowl kick off time 2011″, according to Google Trends the evening before the Super Bowl.”).

Bing is also ranking pages with wisdom like “But a whole bunch of people don’t even know when the Super Bowl starts. As of noon EST today, here are some of the most frequent Google searches, based on the Google Trends site…”

2012 Super Bowl – Everything changes, and the angels rejoice (or maybe that’s just me). Sure, HuffPo is still at it, and still ranking (“For starters, it’s two words, not one. “Superbowl” is an incorrect spelling”), and the Onebox on Google mysteriously goes missing, but the NFL creates a page that is intersection of target audience analysis and organization goals and builds it in such a way that it shows up for searchers.

By the way, that multiple spelling technique isn’t  isn’t just a Super Bowl strategy for the Huffington Post. Take a look at this article about Kanye West’s fashion sense (“He had on a straitjacket (perhaps more commonly spelled “straight jacket”)…)”:

2013 Super Bowl – And this year, the NFL is once again meeting their audiences needs and Google has rewarded them by ranking them first. The Onebox is also back and above the news results this time. And Bing is also providing the start time at the top of results.

Who Should Be Ranking?

What about the organizations other than the NFL that should be ranking?

TV Guide

tvguide.com ranks for [what time does the super bowl start] but not for the slightly more popular [super bowl start time], which is a sign that Google still has work to do in truly ranking based on intent vs. the specific words in the query (and gives HuffPo a little ammunition in defending their choice to sneak in as many query variations as possible). I applaud TV Guide for understanding what their audience is looking for and providing answers, but they could probably do a better job of showcasing other content on the site that would appeal to this audience (right now, they include links in the article, but no images or video that would catch the viewer’s attention and encourage clicks.

CBS

The game airs on CBS this year, but their site isn’t ranking for this query. Amazingly, YOU CAN WATCH THE GAME ONLINE ON THE CBS WEB SITE THIS YEAR (finally, finally, we live in the future), but cbs.com doesn’t rank for those searches either.

My new favorite site, nfl.com, ranks for both (fortunately for CBS, they just link right over to cbs.com). What’s the problem? Same old story. The cbs.com home page links to just a pop up player with the live stream, and doesn’t have a real page available that search engines can index and rank. In fact, the only page on cbs.com that seems to mention the Super Bowl at all is the home page. And without JavaScript enabled, well, the page looks like this:

San Francisco 49ers

The home page does list the start time, but as with teams in years past, that this is the start time of the game can only be inferred. The actual words that people are looking for aren’t there.

Baltimore Ravens

To be honest, I can barely tell that from the Baltimore home page that they’re in the Super Bowl. Sure, they’ve got a hashtag, but this page looks designed for aesthetics rather than based on what users might want to accomplish.

So, it goes. In 2013, large organizations still don’t have the basics down and still aren’t connecting with their target audience as well as they could be.  Knowing what a hashtag is just isn’t enough.

But things are definitely improving. The NFL is getting it right two years in a row. CBS is offering live streaming on your pick of devices. And the latest offering by the Huffington Post is actually about the game, and not an ode to Google Trends:

In 2010, we searched for information. In 2011, we went to Facebook and went to YouTube to see the videos again. In 2012, we talked about commercials on Twitter using hashtags. And, we didn’t replace one activity with another. We mostly added these new behaviors to what we were already doing. So, now we’re searching and Tweeting and Facebooking and YouTubing. What will we add this year? (I’m pretty sure we won’t start QR coding.)

Over the years, more advertisers have included an online component with these Super Bowl campaigns; but amazingly, there have always been a few that don’t seem to know the Internet exists at all. I can’t wait to see what happens this year.

Our Increased Use Of Multiple Screens

According to Nielsen, Super Bowl viewership has grown 26% in the last ten years. Last year, over 111 million of us watched the Super Bowl. Adobe predicts that mobile viewing will double on Sunday.

Multiple Screens

It’s clear that with all of these viewers with multiple devices at the ready, advertisers need to be prepared online as the commercials air. Do their sites rank well in search results for searches for their brands, products, and taglines? Do they feature the  commercials on their sites? As you can see from Google Trends data of some of last year’s advertisers, the search interest is very short term, so advertisers need to make sure they are ready to take full advantage of that interest when it happens.

How Super Bowl Commercials Impact Website Visits

Each year, my analysis finds that commercials cause people to search, which in turn causes people to visit advertiser websites. Adobe found this as well, noting that Super Bowl advertisers see a 20% increase of traffic on game day and that increased traffic levels last about a week (which corresponds to what Google Trends shows in the above graph). As always, I’ll watch this year to see who showcases their Super Bowl ads on their sites (it’s amazing how many haven’t in the past) as that is an obvious way to engage viewers.

What We’re Searching For

Bing has noted that we’re searching for information on the 49ers twice as much as we are on the Ravens. Yahoo has found the same. And according to Neilsen, 49ers.com also has more site visits than baltimoreravens.com.

As always, we’re searching for recipes for snacks to eat during the game. For instance, Allrecipes.com has seen significant spikes in search interest for appetizer recipes in the weeks leading up to the big day. Below, you can see how Blueprint (my company’s search analytics software) is tracking increased search impressions and the number of queries for which Allrecipes.com shows up (which in turn has increased traffic from searches in that category).

What Time Does the Super Bowl Start?

Every year, everyone wants to know what time the Super Bowl starts. In years past, this has been difficult to determine from searching until last year, when the NFL finally started answering the question. As Slate points out, though, this type of search may be less important for companies to rank for as Google provides that answer instantly.

But, value still exists on Bing, where the time is neglected.

And, in case you were wondering, HuffPo is still at it (successfully), as their article explaining all the various ways we search for the Super Bowl start time ranks number one in Google. With an expired image as the main attraction.

(Stock image via Shutterstock.com. Used under license.)

One trend I watch is that of searchers asking what time things start. For any event, you can be sure that searchers will take to Google to find out the start time. I chronicle this every year, for instance, with the Super Bowl. Every year, we search for the start time, and every year, the sites that should have the data either don’t have it all, or have headlines like “countdown clock” until this year when the NFL  built exactly the right page (and I was so happy!).

Yesterday, I was similarly happy when I saw this headline from the LA Times. It’s descriptive and succinct.  It may be based on what people are searching for, but it’s using that information to better understand the audience and meet reader needs.

The article contains valuable information and not just a bunch of words in an attempt to capitalize on search traffic.

Well done, Morgan Little.

PS – this article currently ranks on the first page of Google unpaid results and in the News onebox for the query [when is the inauguration]:

The searches people do that lead them to a site is valuable data to an organization. What other data source do we have that gives us direct access to our customers’ wants and needs? Sure, we can ask them, but customers can lie. Or choose not to answer. By looking at exactly what visitors have searched for, we can learn a lot about who our customers are, what they really want, and if we’re providing it to them.

Search marketers have valued this data for a long time, but it’s useful for many other parts of the organization as well, from traditional marketers to product managers, to support. Shouldn’t everyone want to understand more about their audiences? I think this data is so great, I wrote an entire book about it: Marketing in the Age of Google.

But now, this information is going away. Sort of. Should we freak out? Is there anything we can do to get this data back?

Why Are We Losing Search Data?

First, let’s recap where we are. When someone types words into a search engine, then clicks on a page in the search results, the URL that “refers” the visitor to that page typically includes what the visitor searched for. You can see the referring URL in the site’s server logs, and if you have a Web analytics package installed, it will parse what the visitor searched for from that referring URL and provide it in a handy report.

Just over a year ago, Google stopped including the search term in the referring URL in certain cases. Instead, the referring URL just looked like www.google.com. Web analytics programs didn’t have a way of knowing what the visitor searched for, just that that person came from Google. So Web analytics programs count the visit as Google organic search, but list the search term as “not provided.”

What are these certain cases?

• May 2010 – Google launches encrypted search at a separate URL from www.google.com; you have to go to encrypted search directly to keep your search terms from being included in the referrer.
• October 2011: Searchers on www.google.com who are logged into a Google account are routed through Google secure search. At the time, Google estimated this would impact less than 10% of searches.
• March 2012: Searchers on other Google properties beyond www.google.com (such as international domains) are routed through Google secure search.
• July 2012: Firefox 14 launches, which uses Google secure search for all searches.
• September 2012: Safari in iOS6 begins using Google secure search. Because Google handles secure search differently on mobile devices, this traffic doesn’t show a referrer at all (even that it came from www.google.com), so this traffic shows up in Web analytics programs as direct traffic.
• January 2013: The next version of Chrome (version 25 — not yet officially released, but available for beta and developer use) uses Google secure search for those searching from the address bar. (Those not logged into a Google Account searching from www.google.com will not be routed through secure search.)

Google discussed this latest addition of secure search in the Chromium blog today.

What Percentage Of Traffic Is Obscured By Secure Search?

So, what percentage of Google search traffic comes through as “not provided” and will that total ever reach 100%?

As you can see, how much search data is obscured for your site depends on your audience. Do they tend to be logged into Google when they search? Do they use Firefox or Chrome? Some studies looking at this data represent the shift as a steady increase, and if you follow the line up and to the right, it just keeps going up. But this isn’t a gradual increase, it’s an abrupt change as each source of data shifts to secure search, and then stays flat until the next shift.

For instance, over the holidays, a large percentage of traffic from iPhone users shifted into the “direct” bucket because a lot of people got new phones with iOS6 preloaded.

It’s likely that more data sources will shift to secure search, so no doubt the numbers will increase.

At my company, Nine By Blue, we took a sample set of sites from different industries and with vastly different audiences, to see where things stand right now. No surprise, we found that the percentage of searches reported as not provided has gone up over time. We looked at three dates: December 2011, June 2012, and December 2012 and we segmented US sites from international (English) sites, as secure search was launched initially only the the US:

Below you can see the breakdown for each site as a comparison between December 2011 and December 2012:

For one site (which has a tech audience), 54% of Google organic search terms are reported as not provided, although most sites (particularly the more consumer-oriented) have a much lower percentage of search terms showing up this way.

Should We Freak Out? Can We Get This Data Back?

You should not freak out. Yes, it’s great data and tough to lose, and yes, the percentage obscured is likely to increase, but we will continue to have enough data to make the same decisions and determine the same actions.

One of the most important things we’re losing is the ability to see trends. I always recommend looking at categories of search queries, rather than just at individual terms, but if you look at, say, branded searches, or “how to” searches over time, the trend will likely look as though it’s going down, when in reality, a substantial amount of that traffic has simply moved into the not provided bucket.

Using Google Webmaster Tools Data

One way to get the data back is to look beyond your Web analytics platform. The key issue here is that the search term is no longer in the referrer, but Google still stores the search terms in aggregate in their query logs. They make this data available to you in Google Webmaster Tools. (Disclosure: I helped build webmaster tools originally when I worked at Google.)

You can see the top 2,000 search terms for your site for the last 90 days. Google points out that for 98% of sites, webmaster tools shows 100% of search terms. Of course, the remaining 2% are large sites which are likely the most interested in this data, but even for those sites that don’t see all the search data in webmaster tools, the trends tend to be accurate. So, you can still get a very good sense of what your audience is looking for, and in what topic areas you’re beginning to falter.

I think webmaster tools data is so valuable (see disclosure above about my bias; heh) that my company has built search analytics software that provides insights on top of this data. Blueprint (Disclosure: I sell this product) segments webmaster tools query data into logical topical categories and stores it over time (so you aren’t limited by Google’s 90 day restriction).

Webmaster tools gives you a different view of this data than your Web analytics program does, showing you impression data, ranking, and click through rate for each query. By looking at search queries through this lens, you can tell not only what your audience is interested in, but if traffic drops are due to ranking issues or simply are due to seasonality.

Below, for instance, you can compare these data points for 2012 for branded traffic. Ranking stayed fairly consistent throughout the year, but traffic changes were due in part to search volume changes (because of seasonality) and changes in click through rate from the search results (the site added rel=author markup in September).

Using Paid Search Data

As I mentioned above, it’s only the referral that no longer includes the search term. So, not only do Google’s query logs still have the data available (and used as the source for webmaster tools), but Adwords has this data available as well. If you use paid search, you can get similar audience insights as you can from organic search data. This doesn’t help you if you don’t use paid search, of course.

Using Data From Other Search Engines

Yes, I realize Google likely brings in most of your search traffic. But if what you’re looking for is better understanding of your audience — what they are looking for and if you are providing it — don’t forget that Bing (and Yahoo and the rest) are still sending the referrer in all cases.

Using Web Analytics Data In New Ways

How to Turn (Not Provided) Into Useful, Actionable Data talks about different views of Web analtytics data beyond lists of search terms.

What Does the Future Hold?

Since clearly, more sources are moving to secure search (and it’s possible other search engines may one day follow), it’s important to start making plans now. Take a look at your processes.

Can you get to the same conclusions and action plans with a subset of query data? Can you use webmaster tools and other data to approximate what you really need to make decisions? You can augment audience data with keyword research. You can augment on site behavioral data by looking at a page-based, rather than query-based view. The key is to really think through what you need to accomplish and look at what other pieces of data will help you accomplish that rather than focus on the specific data that’s lost.

Google talked about this in a video they published last year, but it can be easy to forget.

The algorithm changes that Google announces are only a tiny fraction of the changes they are making. And if your site loses traffic on a day that Google announces a change, it’s possible that what Google announced isn’t what caused the drop.

The one question I am asked nearly every day is how can marketers keep up when the algorithms change all the time? And my answer is the same one as it has been since I started working for Google (and hasn’t changed since): Google’s algorithms are always evolving to better reach their goal of providing the best possible results for searchers. Focus on creating pages that are the best possible results for your audience and you’ll always be building to long term success in search.