Walled garden

Facebook has earned its reputation as an AOL-like walled garden by keeping most of the content generated within Facebook hidden from those who haven’t logged in.

Not only does a login screen stop general Internet users from viewing Facebook content without logging in, it also impedes search engine indexing.

Currently, Google can index:

• a limited view of Facebook user profiles
• “Fan” Pages, including user posts
• Facebook comments left on third-party sites

At one point in time, search engines were also able to index many Facebook groups, but this does not appear to be the case any longer. I should also note that users may change a privacy setting so search engines can’t index their specific profile.

Way back when Google offered real time search, Google was denied access to Facebook user posts, an advantage Bing had thanks to a special agreement certainly facilitated by Microsoft’s investment in Facebook. Facebook user posts weren’t otherwise available to search engine crawlers.

During research on another topic recently, I happened to discover that posts from Facebook user profiles are now finding their way into Google search results.

Oh my, Facebook user posts in Google search results (click to enlarge)

The same is true for Yandex; Bing’s normal Web search didn’t return any user posts. Recent Facebook and Twitter posts are still available in a few countries via Bing’s special social search. To be clear, the number of Facebook user posts Google is indexing appears to be small.

User profiles still don’t display posts for users which aren’t logged in, so how is Google finding Facebook posts to index?

The posts I looked at had not appeared on Facebook Pages, which do display posts even if a user isn’t logged in. Groups appear to be off-limits. Facebook offers an XML sitemap to search engines, however access appears to be limited by IP so I cannot actually verify what it contains.

A Facebook user profile seen as an anonymous user doesn't contain posts (click to enlarge)

The most likely answer is that Google is picking up some Facebook user posts which have been explicitly linked to from other sites, like those also automatically posted to twitter. If the original Facebook post is too long for Twitter, a fb.me link connects to the original post on Facebook.

At the moment, users don’t need to login to Facebook to access a public user post as long as they have the direct URL. But you never know if or when Facebook may put user updates back behind the walled garden.

A few Florentines, mostly former colleagues from the Italian internet company Dada, set out in April 2011 to develop a new search engine, Attrakt, which would focus specifically on hosting specialized search engines curated by the Web community.

Attrakt’s Index Relies On Curator Contributions

As such, Attrakt isn’t a generalist search engine, with a broad index ready for everyday searches. Attrakt’s index is primarily seeded through links specified by curators of custom search engines, what Attrakt calls boxes.

As in the early days of Wikipedia, if you’re lucky enough to search on a topic someone has already curated, then you might be impressed. But for the most part, Attrakt is fairly empty, a problem Attrakt partners Andrea Dotta, Luca Ciavarella and Gabriele Miceli acknowledged won’t be resolved until Attrakt reaches a critical mass of contributing editors.

Putting Attrakt To The Test With Schema.Org Rich Snippets Use Case

Users can explore Attrakt’s potential by searching or creating a customized search engine on a topic of particular interest; the Attrakt team mentioned travelling in Italy and 80s synthpop among many examples. These two illustrate the use multimedia in addition to textual resources. I decided to try a search marketing use case: the Schema.org semantic web markup standard defined and supported by Google, Bing, Yahoo and Yandex.

By implementing Schema.org standards, websites can enable the display of many types of enriched search results, what Google calls rich snippets. Enriched results can be very beneficial for Web marketers and users alike but there’s a lot of material to wade through.

There’s the official standard, supplemented with separate documentation from each search engine. Extensions are available for many CMS‘ including WordPress, Joomla, Drupal and IBM WebSphere. Google, Bing and Yandex each offer testing tools.

Armed with around 60 selected links to pages and sites regarding schema.org and rich snippets, I created a new Attrakt box, or specialized search engine. Searches using the resulting Schema.org Attrakt box provide answers from these curated resources.

Creating an Attrakt custom search engine is a fairly straight forward process. An editor needs to provide a name for their engine, assign a category and country and add a few tags.

Each link is then specified together with a few attributes such as a link category (e.g. Blog, News, Tool, Official) and whether Attrakt should index just the URL or the entire site. The link categories allow an end user to browse through the links assigned to each of these categories.

Overall, the creation interface could be simplified: entering 60 URLs, one by one, is a tedious process. It would be quicker to paste all the URLs into a single form, assigning categories in a successive step. The box country should be optional. The topic of many curated search engines, including this one, will transcend geography.

Link categorization is also problematic: should an official Google blog post on Schema.org be tagged as Blog, News or Official? While at first glance blog might seem to be the best answer, a Google Blog Post is really an official company statement which in of itself is often a news item, which happens to appear on a blog publishing platform. Clear?

Once a custom search engine box is created, it can be modified or deleted, although the use of grayed out text for the edit link both makes it hard to find and implies it isn’t available as a function.

To be fair, the Attrakt team is continuing to refine both the user interface and the underlying search engine technology. By the time you read this article, much of what you see will undoubtedly be improved. In the future, it will apparently be possible to assign a link to multiple categories. Attrakt is also working with researchers at the University of Florence on algorithmic solutions.

Once a box is created, it can be shared with others via a link. Attrakt will periodically crawl a box’s URLs to check for changes. Box curators will see a message display when Attrakt detects changes. It would be nice if they could also receive notifications via mail or RSS, similar to the way Google Alerts works. Search box curators will be able to embed their Attrakt search box on other sites, a feature planned for May 2012.

Are Curated Search Engines Really Useful?

A curated search engine potentially has two specific advantages over a generic search using Google, Bing or Yandex. The first is it will exclude content which strictly speaking answers the query but is of low value, like the plethora of parrot posts which appear just after a Google blog post is published. It often seems that 9 out of 10 repeat the official news without adding any additional insight.

A second benefit is to surface specific content which might not have otherwise showed up in a generic search. At the time of this writing, Bing’s Schema.org testing tool is poorly indexed in Google, but can be found browsing the Tools links of the Attrakt Schema.org search box.

Attrakt Wants Crowd Sourced Curated Search To Go Social

The quality of a custom search engine depends on the ability of the curator and on the technology providing the search results. Currently, Attrakt will display the user name of a box curator, but there’s no way for a casual user to determine if the curator is authoritative or not.

Attrakt currently collects profile information during the user registration process; it would be nice if they would also display profile information together with other box attributes. Such display would also give search box curators a greater incentive to contribute quality search boxes to Attrakt.

Attrakt highlights some hot search boxes on their homepage, but it isn’t possible to search for a search box on a topic nor is there a directory of search boxes.

In some cases, search boxes are listed next to results in a general Attrakt search, although there are some geolocalization issues still to work out. Attrakt primarily relies on a search box’s author to promote a specific curated search engine.

Attrakt search boxes are curated by a single user. It would be interesting to allow collaborative search box editing, similar to Wikipedia’s support for multiple authors. In a world filled with spammers, this isn’t so simple to successfully implement.

Attrakt plans on adding a robust social layer – a layer which will undoubtedly make Attrakt much more attractive to curators and search users alike.

On this point, Attrakt says:

Users will soon be able to follow each other and each others’ boxes, create elaborate user profiles to declare their areas of expertise. A Twitter stream will soon help animate search boxes with up to date content. Boxes already have static URLs for sharing, and sharing will be encouraged by share buttons.

Outside of the Attrakt platform, we’re planning on using twitter to drive traffic to peoples’ search boxes using relevant hashtags, and if a twitter account is present in the user’s profile we’ll tag that user.

Attrakt Is Still In Early Stage Of Development

A reader would be forgiven if they have the idea that Attrakt is very much a work in progress. The Attrakt team began armed with a dream and a modest €50,000 in self-financing. They’ve come a long way in a year and attracted an additional €500,000 in funding but are still months away from realizing the product they have in mind.

Mindful of this, they’ve eschewed seeking out press coverage, preferring to release and refine features out of the limelight. However, a site focusing on Italian news tipped me off to this new source of traffic, so now you know about Attrakt too.

Services Similar To Attrakt

Google’s customized search is probably Attrakt’s closest competitor. Many websites have added a basic version of it to supply a quick and easy site search, although it also includes features like Attrakt’s link categories, what Google calls refinements.

Perhaps the primary difference, beyond Google’s algorithmic prowess, is that Attrakt says its embedded search boxes will be advertising free (Attrakt does carry advertising on their site).

Attrakt also facilitates the sharing of curated custom search engines and plans on further differentiating themselves with additional social features. Readers can compare the search results for the same Schema.org resources indexed by Attrakt and Google.

Mahalo has positioned itself as a human edited search engine, but they don’t crowd source their contributions.

Delicious is by far the most used collaborative bookmarking service, however they don’t actually index the content of the saved bookmarks.

Volunia is a social search engine also from Italy, however their focus is more general.

Attrakt Opportunities For Marketers

Attrakt offers marketers the opportunity to demonstrate their personal or company expertise in a particular subject area through the creation and promotion of a curated search box.

As with all things social, the rules of the game have changed. The most successful search boxes will include comprehensive coverage of a subject domain, even if that means including resources from competitors.

What Webmasters Should Know About Attrakt

Attrakt robots.txt Support

Attrakt supports the robots exclusion protocol, known as robots.txt. Attrakt’s crawler is called attrakt. If Attrakt is crawling the site, web server logs will contain the user agent:

Mozilla/5.0 (compatible; attrakt/1.0 +http://www.attrakt.com)

Meta tags are not yet supported, nor is there support for sitemaps.org and schema.org protocols.

Keyword Tracking In Google Analytics And Similar Systems

Attrakt provides search query information in URL referrers using the defacto standard q=<query> URL parameter name/value pair.

Users of most digital media measurement systems will need to add recognition logic for Attrakt searches, otherwise Attrakt will show up as a simple site referer. Google Analytics asynchronous tracking code users can add ['_addOrganic','attrakt.com','q'] to their tracking code.

The story behind the Attrakt name? When asked, the Attrakt team demurred. Its origins seem to be lost in the sands of time.

Volunia founder, Massimo Marchiori, has said that one of the main reasons to launch Volunia now was to gather feedback from beta testers. I spoke with Marchiori to find out what Volunia has learned after a week of exposure to users and to understand where Volunia is headed. I also asked him about Volunia’s support for webmasters.

Volunia Users Say Search Result Quality & UI Need Improvement

About 400,000 users have signed-up to use Volunia of which, 100,000 were provided access in the first week. Access is being doled out slowly in order to manage server loads. The two primary criticisms leveled against Volunia by users have been regarding disappointing search results and, especially from design conscious Italians, the user interface.

Marchiori told me that the greatest problem with the search results is that Volunia has only indexed a small sample of the Web – about 1 million sites or 1% of all the websites worth indexing. The open directory, Dmoz, was used to seed the initial list of sites to crawl. Additional sites deemed important in their home countries were manually added to broaden international coverage.

Volunia planned to focus more of its initial resources on their social network features as that is where they believe to be providing the greatest innovation. As a result of user feedback, Volunia will also be working to improve the search engine index coverage sooner rather than later. The Volunia team, currently 10 people, some of whom only recently came on board, will also be revisiting the user interface.

A Solution Is On The Way For Incompatible Sites

Many users also noticed that some sites, including Google properties, Facebook and even Search Engine Land, are incompatible with Volunia’s social features menubar. Some sites reload without Volunia’s menu and others just doesn’t load.

In some cases, the behavior is deliberate on the part of the site owner – they don’t want others framing their content. Volunia is studying enhancements to their browser toolbars, available for Firefox, Chrome and Internet Explorer browsers (and found in the Addon menu), which should resolve these problems. For now, site owners could update their anti-framing logic to whitelist Volunia.

Friendship Crisis: Too Few Volunia Friends

Volunia users start with clean slate of friends, akin to leaving Kansas for Los Angeles or San Francisco to reinvent oneself. Many people have told Volunia that they want a way to upload their existing contacts, a feature Volunia aims to make available sooner rather than later.

Volunia does offer a people search function, but its usefulness depends in part on the completeness and privacy settings of a user’s profile. A people search based on email address, a little known Facebook feature, would be a nice touch.

What Webmasters Need To Know About Volunia

Figure 1: Visual Navigation map for a section of Wikipedia, including Volunia Social Network Menubar, Site Information & Side Chat

Site Authentication

Volunia offers website owners the possibility to authenticate their site(s), similar to other search engines. Once this is done, the visual site navigation maps presented to Volunia users can be personalized by adding, moving, and / or deleting pages. Page names can also be changed. These names also appear in the Volunia chat window title – a point worth keeping in mind.

In the future, it should be possible for a site owner to regenerate the visual site map at different zoom levels should they want to revert any personalization. This functionality has apparently been developed but it hasn’t yet been enabled as Volunia prefers to ensure the features already deployed work well under increasing user loads before adding additional server overhead.

Volunia allows multiple users to authenticate a site. Each user who verifies a site will be added to the site information visible to users. No mechanism yet exists to revoke a user’s permissions – a point worth considering when deciding who can authenticate a site.

Site Submission

Modern search engines use links to discover and rank sites, so it isn’t normally necessary to submit sites to a search engine. Volunia site authentication will also add a site to a queue of sites to consider crawling, a type of site submission. This feature is apparently already in place but not yet enabled behind the scenes.

Site Promotion Tools

A share option within Volunia’s menubar allows users to promote a page they like on Facebook, Twitter, via email or by a link they can paste wherever they like. In an initial test, a link shared on Twitter brought me to Volunia’s home page, not the page I wanted to share. I was actually trying to determine what type of redirection was happening: i.e. will the shared links receive appropriate social signal credit from Google and Bing?

Volunia doesn’t yet offer an endorsement feature similar to Facebook’s Like and Google’s +1 but Marchiori acknowledged that a social service won’t be complete without one.

Volunia offers several plugins which allow site owners to embed Volunia features, such as the visual navigation map, directly into their sites. Unfortunately, as Volunia doesn’t allow anonymous users in the current test phase, the plugins will display a login form rather than the Volunia content. Its probably a bit premature to consider using these.

Robots.txt Support

Volunia supports the robots exclusion protocol, known as robots.txt. Volunia’s crawler is called Voluniabot.If Volunia is crawling the site, Web server logs will contain the user agent:

Mozilla/5.0 (compatible; Voluniabot/1.0; spider@volunia.com)

Meta tags are not yet supported, nor is there support for sitemaps.org and schema.org protocols.

Keyword Tracking In Google Analytics & Similar Systems To Come

In the era of Google’s questionable decision to start reporting search query keywords for authenticated users as [not provided], I asked Marchiori about the possibility to supply user search query terms in URL parameters which could be recognized by existing web analytics systems.

He said they hadn’t initially planned on doing this as they thought they’d provide their own feedback to webmasters, but he agreed it would make sense to support compatibility with existing third party digital marketing measurement tools. On a related note, see how to track search query keywords from blekko.

Perhaps Volunia Could Better Manage User Expectations

During the launch presentation, Marchiori said that Volunia doesn’t aim to be a Google competitor – it sees itself as complementary to other search engines. The more I look into Volunia’s social network layer, the more I understand what he means.

For those who want to use Volunia’s social network features, i.e. interacting with others who are visiting a website page, but aren’t yet happy with Volunia’s search results, Marchiori told me they can use another search engine together with Volunia’s browser extension which allows a user to open Volunia’s menubar in any browser tab.

In my testing, I found a searcher can use Bing or blekko together with Volunia’s social menubar. Neither Google nor Yandex are usable due to the frame incompatibility issue mentioned earlier.

In our discussions, it was evident that Marchiori has a clear vision for Volunia’s future development and that he seems very keen on addressing issues raised by user feedback. I’m optimistic that Volunia can do well, providing it is able to attract the resources it will need to continue investing in product development.

Volunia has raised an initial €2 million in funding – an amount which pales in comparison to the $24 million in funding blekko had received by its launch, not to mention the additional $30 million blekko has received since. Potential competitor RockMelt has received $40 million yet their service only works in one browser which isn’t available for all computer platforms.

By announcing a beta test and implying power users would receive sign-in credentials on the launch day, Volunia set itself up for some criticism which probably could have been avoided. Volunia could have been better positioned as a working prototype in an early alpha test. Social media channels could have updated potential users regarding the progress of credentials distribution.

Now that Volunia has been able to unveil its vision for social search and social Web navigation, it should be better positioned to attract the additional funding necessary to increase server capacity, add to staffing and address external communications.

1. High level site previews in search results
2. A multimedia search within a site function
3. A social layer which, among other things, allows Volunia users to share information and connect to one another

My sense is that it is the social layer which will be most appreciated by Volunia users. Let’s look at each.

Volunia Search Result Previews Offer A “Fly-Over” Site View

During his launch presentation, (in Italian, starts at 40 minute mark) Volunia founder Massimo Marchiori described search users as similar to chickens, trapped in cages and incapable of flying.

Users have been forced to choose search results by consulting one of those classic 10 item title, summary and link lists for too long. Wouldn’t it be nice if users were freed from their cages, and not only, were actually able to fly over a site, viewing a visual or a grid map, before committing to visit it?

Volunia, perhaps from volare, to fly, offers two types of high level site map previews, potentially freeing users from commitment tyranny. The first type, a visual map, aims to group areas of a site together in neighborhoods.

Figure 1: Volunia search result site fly-over: visual map

The second site preview map is in the form of a grid, reminiscent of computer folders. The expectation is that this format will be more useful to people searching from devices with small displays, like smartphones.

Figure 2: Volunia search result site fly-over: grid map

Interactive drill-down versions of the maps are also available from a Volunia menu bar which is visible while navigating a site. Site owners can improve the maps using a Volunia provided sitemap editor.

Figure 3: Volunia sitemap editor

Result previews aren’t exactly a new concept – Ask.com introduced their binoculars feature in 2004.

Where Volunia differs is in their choice to show a site map preview instead of a page preview. Many searches are indeed navigational in nature, one reason Google provides their sitelinks for some queries.

Volunia may be on to something.

Volunia Wants To Surface Multimedia Otherwise Hidden In A Site

Initially the primary focus in Web search was on textual documents, particularly the html kind, rich in semantic structure with their glorious title, heading and paragraph tags.

Oh, I didn’t mention the links between documents, did I? Other Web content formats, from PDF files to images and then video posed much greater obstacles to search engine indexing for a number of reasons.

Today, major search engines like Google offer navigation links to enable a user to search just images or video.

Ambitious searchers can usually find an advanced search syntax page which allows them to limit their searches to specific sites and file types, but for the most part major search engines have taken the “don’t make me think” approach, providing searchers with a blend of media types in search results, what Google calls universal search.

Volunia on the other hand wants to make it easy for a user to discover the multimedia richness hidden in sites like NASA by providing a very visible multimedia site search filter.

Figure 4: Volunia's multimedia search filter

The Web Has Come Alive, Says Volunia: Volunia’s Social Layer

It isn’t immediately clear how useful this feature will prove to be: after all, even if we want to follow the “wisdom of the crowd”, there’s no way to know if the site’s current visitors from Volunia are actually happy with their choice, nor would it be clear to what extent one searcher’s expectations for a page align with those already visiting that page.

The same visitor counts are also available as layers on the site navigation maps.

Seek & Meet: Interact With Other Volunia Users

What might arguably be Volunia’s greatest innovation is in letting fellow search travelers to a page interact with each other, what Volunia calls seek and meet, a feature which feels very reminiscent of Google’s now closed Sidewiki, albeit with two key differences. The first is that users can interact in realtime.

Figure 6: Volunia seek and meet interactive site chat

This birds of a feather real time information sharing might prove useful in a number of situations where people are looking for pre- and post- purchase information.

In the pre-purchase phase, a searcher might want to interact with other users to better understand the product or service they’re considering, not to mention to discover what alternatives others are considering.

Sometimes, it’s nice just to have confirmation that we’re making the right choice. In the post-purchase phase, searchers might be able to resolve support issues by consulting with other searchers – potentially reducing a company’s support costs while providing interactive peer to peer support 24 hours a day.

Each Volunia user is able to fill in a personal profile, much like any social network. The matchmaking possibilities are clearly endless, but I suspect it would be best if I don’t go there….

It isn’t rare to see glowing online reviews written by someone with a connection to a product or service, and equally harsh reviews from competitors or ex-employees with an ax to grind. It doesn’t take much to imagine people attempting to scam the system by introducing fake search users to interact with other searchers.

It remains to be seen if “seek and meet” really is something people will take to. Real time search collaboration will well depend on a critical mass of socially oriented searchers congregating on the same sites at the same time, no easy feat for a niche search engine.

Site owners will undoubtedly be pleased with the second apparent difference to Google’s Sidewiki: commenting can be disabled if desired, something Google didn’t allow.

Volunia, The Company, And A Few Volunia Tidbits

According to data published by Italian business paper Il Sole 24 Ore, Volunia was founded in 2008 by Massimo Marchiori and entrepreneur Mariano Pireddu, with Pireddu providing €2 million in funding to date. You might not immediately recognize Massimo Marchiori’s name, yet as an academic Massimo has been working on the theoretical issues of Web search for years.

His seminal 1997 paper, The Quest for Correct Information on the Web: Hyper Search Engines, would serve as one of the sources of inspiration for two Stanford students, Sergey Brin and Larry Page, who would acknowledge Marchiori’s contribution to their own work, Google.

During Volunia’s beta phase, over 100,000 people are being invited to become “power user” beta testers. The Volunia user interface is in 12 languages but Marchiori said during the launch presentation that the actual index coverage isn’t limited to those languages.

The Volunia team has ideas for Volunia “extensions”, i.e. new functionality, which will be added to the core, the hard part which has already been done. Advertising will be added to the service.

Marchiori Says Social Needs To Emerge In Search

In an introductory video, Massimo notes that Volunia stems from an idea he’s harbored for several years, an idea for a “different perspective of what the search engine of the future should be”. In the Il Sole 24 Ore report Massimo said “The Web is a living place, there’s information, but there’s also people. The social dimension, already present, just needs to emerge“.

Bing, which started incorporating social signals from Facebook in 2010, and Google, which launched its social search in 2009, would probably both argue that the social dimension to search has already emerged. So would upstart blekko and to a lessor degree, the Russian Yandex.

Armani, Chianti, Ferrari… And Volunia

Volunia is based in Italy, not in Silicon Valley as one might have guessed. Italy actually has a history of search engine excellence. Google may well power most Italian portals today, but the talent behind a now defunct Italian search engine, Arianna, led Ask.com to locate its European R&D headquarters in Pisa.

The Reality Check: Search Isn’t Easy: Volunia Faces Many Challenges

The basic task of a search engine, finding, indexing and retrieving the world’s information, is a complex one. The size of the Web is immense. There’s the problem of searcher intent: we know what we’re looking for when we type a brief search query, but those few words are often open to multiple interpretations.

Google has conditioned searchers to expect lightning speed and mostly fresh and relevant results. Volunia will have to perform well on all these fronts and navigate thorny issues of privacy.

Many start-ups have nonetheless tried to compete with Google and Bing. Some, like Cuil, ran out of funding before gathering significant market share; others like blekko, with far greater funding, are still working hard to win over hearts and minds. Whether Volunia will be able to pull this off remains to be seen.

Kick The Volunia Tires Yourself!

Volunia has a sign-up form for those who want to try it out. Go kick the tires and support the underdog! From messages that I’ve seen on Friendfeed, Twitter and Facebook, very few have actually had a chance to actually use Volunia, credentials are only dribbling out, most likely in an attempt to avoid problems similar to what Google faced when they first opened Google Analytics to too many people at once. I based the considerations made (and images) in this article on demo videos released by Volunia in order to give you a preview of what to expect.

It is possible to properly attribute blekko search traffic referrals by implementing advanced Web analytics configuration techniques.

Is the extra effort worth the trouble? Certainly investors are betting on blekko. In late September, blekko received an addition $30 million investment, half of which was from Russian search engine Yandex.

Why Web Analytics Systems Can’t See Blekko’s Search Keywords

The underlying difficulty many Web analytics systems encounter in recognizing blekko as a search engine is due to a reliance on search engines to specify the search keywords in the request for the destination webpage by using parameters in the referrer URL.

Unfortunately, blekko deviates from what has historically been the de facto standard for transmission of search query information. In the absence of advanced tool configuration, traffic from blekko will show up in a referring sites report instead of an organic search traffic report.

While this scenario is better than nothing, it makes keyword analysis difficult. Knowledge and analysis of search query keywords is a key element of search marketing as a user’s search keywords express their intent when selecting a site from a listing of search results.

In the following discussion, Google Analytics will be used to illustrate solutions, but the same data transformation logic can be applied to other tools as well. The technically inclined are encouraged to read on. Everyone else might want to pass this article on to their technical staff for analysis and implementation.

Deconstructing & “Fixing” The Blekko Search Referrer

A typical blekko referrer URL contains blekko’s domain name, the parameter ws and the user query. Spaces between multiple query words are replaced with plus signs.

In the following example, the user’s search keywords stop+blekko+indexing follow http://blekko.com/ws/ in the search engine results page URL, visible in the browser address bar.

Figure 1: A blekko search result with user keywords in the result URL

Unfortunately, this isn’t in the form most Web analytics systems were designed to recognize when processing organic search traffic.

The URL piece ws/ needs to be expressed as a parameter, in the format name = value, i.e. ?ws=stop+blekko+indexing.

Thus: http://blekko.com/ws/stop+blekko+indexing needs to be transformed into http://blekko.com/?ws=stop+blekko+indexing

Two Possible Solutions To Solving The blekko Attribution Problem

There are two potential solutions to proper recognition of blekko as a search engine by Web analytics systems. The first approach is to modify the raw data sent to the Web analytics data collection system. The second is to transform the data collected by the Web analytics system before it is processed for reporting.

Option 1: Modify The Search Referrer Sent To Google Analytics

The blekko referrer URL needs to be transformed into the de facto standard format which includes the user query using URL parameters, e.g. ?query=keywords, a format recognized by most Web analytics systems. JavaScript is well suited to this type of task.

The HTML standard doesn’t allow direct manipulation of the document (page) referrer, but the referrer sent to the Google Analytics data collection system can be changed by assigning the modified value to a special parameter, _setReferrerOverride.

In the asynchronous Google Analytics tracking code which follows, the code in bold will transform blekko referrer URLs and assign the result to _setReferrerOverride.

The code:

• assigns the page referrer to the variable blekref
• tests to see if blekref contains the string http://blekko.com/ws/
  • if so, tests to see if http://blekko.com/ws/already contains URL parameters, which start with a ?
    • if so, substitutes the ? with a & which indicates successive URL parameters
  • substitutes ws/ with ?ws=
  • places the modified URL in the Google Analytics variable _setReferrerOverride

<script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount','UA-xxxxx-1']); var blekref = document.referrer ; if (~blekref.indexOf('http://blekko.com/ws/')) { if (~blekref.indexOf('?')) { blekref = document.referrer.replace('?','&'); } blekref = blekref.replace('/ws/', '/?ws='); _gaq.push(['_setReferrerOverride',blekref]); } _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script>

Note: this code will need to be modified if blekko changes their URLs, such as using www.blekko.com in addition to blekko.com.

As with any code changes, do be sure to consult current Google documentation and fully test changes before putting them in production. JavaScript errors in a page can stop successive JavaScript from being executed. No warranties are provided, implied or otherwise!

Tell Google Analytics How To Recognize blekko As A Search Engine

Once blekko’s referrer string has been transformed into the standard query parameter = keywords format, the second and final step is to add blekko to Google Analytics’ list of known search engines.

This also has to be done within the Google Analytics tracking code inserted in each page of the website.

The syntax to add a search engine is of the form:

_gaq.push(['_addOrganic','blekko','ws']);

where blekko is a string in the referrer domain and ws is the query parameter name.

This has to be inserted somewhere before the ['_trackPageview'] call, i.e. from the above example we can change

_gaq.push(['_trackPageview']);

to

_gaq.push(['_addOrganic','blekko','ws'],['_trackPageview']);

A more in depth discussion on working with the _addOrganic configuration option can be found in an article on Google Analytics search engines list customization.

Tracking Page Speed In Google Analytics?

During testing, it emerged that Google Analytics would ignore the value of _setReferrerOverride if _trackPageLoadTime was specified.

Using the Google Analytics debugger, it became clear that Google was deprecating _trackPageLoadTime so the message is simple: remove _trackPageLoadTime from existing tracking code to avoid unforeseen consequences.

Option 2: Transforming Collected Data

Should sites not be able to modify their Google Analytics tracking code, or should their Web analytics system not provide a referral override option, there is a second approach which should work as well. The technique involves transforming the collected data before reports are created.

In Google Analytics, Profile filters perform this task. Filters are a wonderful, but underutilized, Google Analytics feature and this is a great example to show off their potential. The process is:

1. Identify search referrals from blekko
2. Update the medium, source and term campaign fields accordingly.

Unfortunately, Google Analytics only allows a profile filter to update one field at a time, so it will be necessary to create multiple filters.

To do this, enter the Google Analytics administration area and create new advanced custom profile filters as illustrated below. Filters can be created at the account level, making it easy to quickly apply the filters to multiple profiles.

Figure 2: Blekko set campaign medium filter.

The above filter selects referral traffic from Blekko which contains /ws/ in the URL and changes the Google Analytics Campaign Medium from referral to organic (required for blekko traffic to appear in the organic search report).

Currently, it isn’t strictly necessary to include the Field B requirement, but it may be helpful in order differentiate from paid search should Blekko offer it in the future. Note that the values referral and organic must be in English, even if the Google Analytics interface is in a different language.

Figure 3: Blekko set campaign source filter

Google Analytics will automatically identify blekko.com as the source of traffic, however, some may want to see the source report as “blekko” instead of “blekko.com” in their reports, so as to be consistent with the other search traffic sources. The above filter, which is optional, will do just that.

Figure 4: Blekko set campaign terms filter

The set campaign terms filter captures the user’s search query and places it in the Campaign Term field, the source of the keywords which appear in the Google Analytics organic search traffic report.

The key is in the regular expression specified in Field A:

^http://blekko\.com/ws/(.+)\??

It says, select referrers which start with http://blekko.com/ws/, capture everything that follows up to a question mark, which may or may not be present, and put the result in the campaign terms field.

Astute readers might note there is still one problem remaining: multiple keywords will still be separated by one or more plus signs which should be converted to spaces.

Unfortunately, it doesn’t seem that Google Analytics supports global (multiple) substitution using filters, but in the spirit of crowd sourcing, a reader may want to chime in to improve this last step.

One work around might be to create multiple substitution filters which will replace a single + with a single space:

Figure 5: Filter to substitute one plus sign, +, in blekko search keywords, with a space

Once the profile filters are created, they need to be activated by apply them to each Google Analytics profile.

The order in which the filters are applied is not important, with the exception of the plus sign substitution filters, which must come after the set campaign term filter. A Google Analytics best practice is to keep at least one profile without any filters as once data has been transformed by filters as there isn’t any way to revert filter changes.

An unadulterated profile is also helpful in troubleshooting any issues which may arise. New filters can (and should) be tested on a new “throw-away” profile before being applied to production profiles.

A Note About blekko’s Slash Tags

One of Blekko’s distinguishing features is the ability, encouraged by blekko, of users to further restrict their search queries using search operators and filters, what blekko calls slashtags. As the slash tags do condition the user’s query, and to a degree express the user’s intent, neither of the approaches illustrated above attempts to remove slashtags, if present, from the search query keywords.

In June 2011 Google began to pilot a way for content authors to associate a photo and an author byline to the display of their articles in Google search results. Google extended the pilot to Google News searches in November 2011.

The addition of author information in search results continues Google’s support of microformat html markup to enrich a search result, a concept originally pioneered by Yahoo as enhanced results, called rich snippets by Google and known by Bing as Tiles.

Why Authorship Markup & Rich Snippets Are Worth The Effort

Rich snippets offer websites the chance to stand out from the crowd in search results, enticing more clicks in the process. Authorship markup is particular visible due to the presence of an author’s photo – humans are wired to gaze at photos of people.

The combination of an author photo and byline can add credibility to what might have been just another otherwise anonymous search result.  A domain name in a result listing can pull just so much weight.

Authorship Rotates Around The Author Profile / Hub

Google’s implementation of author rich snippets requires an author profile page containing an author head shot photo. The photo will appear in the rich snippet; originally it was positioned on the right, before assuming a spot on the left.

Google also creates an author byline which links to the author’s profile page:

Figure 1: Author rich snippet in Google search results contains author photo and byline. This example also contains a count of Google+ followers.

Google has many sources available for author hub pages. Websites which contain high quality content, including news outlets, usually have their own author profile pages which, by virtue of association, will be as authoritative as any of the content by that author on that site.

Content authors may also manage their online digital identity by promoting their profiles on one or more social networks, from the business oriented LinkedIn to the more general Facebook, Twitter and Google+.

In its authorship markup pilot, Google has chosen to support just one authorship profile hub, Google Profiles, which is just a click or two away from what appears to be their real objective, the promotion of Google+, discussed in more detail below.

Google’s Authorship Markup Isn’t Search Engine Agnostic

Google’s choice to limit author profile options to their own Google Profiles is rather troublesome as content authors must actively promote an external profile to the detriment of their own profile or a profile on an independent site of their choosing.

Promote? Indeed. Google requires links to the Google / Google+ Profile from the author’s website, with no mention of provisions for using the infamous rel=”nofollow” link attribute or the less invasive <link> html tag.

As other search engines will be indexing this Google specific code too, Google Profiles have an unfair advantage in search results not just in Google but in those Google competitors as well.

Google also promotes the author’s profile page as a byline link included in search results for the author’s content. Instead of a search engine user clicking directly to the author’s site, a user can easily be diverted to another Google property, Google+.

On a social media site, the goal is to use engagement to keep a user from leaving, not too unlike Charlie and the MTA.

Given Google’s effective search monopoly in many of the markets in which it operates, its hard not to interpret Google’s promotion of Google Profiles/Google+ as a not so subtle conflict of interest. What is truly surprising is that this behavior is so uncharacteristic of Google.

Adherence To Industry Standards? Maybe, Maybe Not

In initial announcements, Google touted their adherence to industry standards in designing the authorship rich snippet markup specification. They noted the use of rel=”author” (credited to html5, it isn’t actually new) and the XHTML Friends Network rel=”me”.

Google even said they support article content markup specified by the search engine industry sponsored Schema.org project (like sitemaps.org, schema.org was conceived to allow websites to implement functionality compatible with all major search engines).

Yet as long time Microsoft watchers will know, the devil in the details. Google’s authorship markup documentation omits any mention of the Schema.org standard and implies a requirement to use a Google/Google+ Profile, a detail clearly not part of the schema.org specification.

Perhaps one of Google’s authorship markup showcase examples does use Schema.org markup – the curious will have reverse engineer each example in a quest to find the answer.

After the first few months of the authorship markup pilot, Google also said they would support non-standard ways to link content to Google Profiles. The first change was to allow the rel=”author” anchor tag attribute to be specified as a URL parameter ?rel=author instead.

Presumably, this was done to facilitate implementation in CMS systems which don’t allow easy specification of anchor tag attributes. Unfortunately, this variation breaks html5 and Schema.org standards, an important point Google failed to mention.

When Do Authorship Snippets Show Up?

Once an author has diligently implemented authorship markup and tested it using the Google rich snippets testing tool, Google will need to recrawl existing articles before an author’s photo and byline can conceivably show up in search results.

In reality, Google is still piloting authorship markup; in November 2011 Google finally added the phrase “Google doesn’t guarantee to show author information in Google Web Search or Google News results” to their documentation. Translated, that probably means that only “authoritative” sites, those with lots of quality incoming links and other signals like frequent updates, are candidates for consideration, and only for searches in English on Google.com.

Unfortunately, in their promotion of authorship markup Google is not at all clear on this point, setting up false expectations for many of those who invest in authorship rich snippet markup implementation.

Judging from Google’s deployment of other rich snippets, Google will most likely become more permissive over time as to who they recognize with authorship snippets and authorship snippets will eventually be rolled out internationally.

The Prominent Promotion Of Google+

After a cursory reading of Google’s authorship markup documentation, its easy to come away with the impression that Google requires authors to have a Google Profile as it in some way will allow Google to better trust an author’s content.

Yet, there is nothing in the current Google Profile model (that without Google+ integration) which intrinsically vests a Google profile with any greater authority than an author profile of an author’s choosing. It seems that Google has a not so hidden agenda to promote Google+, promotion which manifests itself in multiple ways:

• Google’s authorship markup documentation says to use the Google Profiles icon, to link to the Google Profile. What does g+ have to do with a plain Google Profile, one asks? As an alternative, Google illustrates the use of the anchor text Google+. This can be changed, as long as the + remains.  Yet if rel=”author” information is already included in a link, there appears to be only one conceivable justification in requiring the use of a + in the anchor text: promotion of Google+ since many will inevitably use Google’s suggested Google+ as their anchor text.  The link Google uses in their documentation example? plus.google.com, not profiles.google.com.

Taken together, its no wonder that a casual observer can be forgiven for confusing Google Profiles with Google+ profiles. A purist could create a Google Profile not linked to Google+, but Google isn’t going to go out of their way promote this option any more than Facebook promotes its little known Business Account.

• Google is mum on the use of rel=”nofollow”, which should be a requirement given that authors are linking to their Google Profile through not so subtle coercion: they need to maximize their visibility in the search engine which has an effective monopoly in many of the markets in which it operates.

• Google does not allow authors to promote their own, arguably more authoritative, author profiles. Or maybe Google does as they did say in their initial announcement that they support schema.org markup, but there’s no word on this nor an example in Google’s authorship markup documentation.

• Google does not explicitly allow authors to specify alternative profile pages on competitor sites like LinkedIn, Twitter or Facebook. None of these profiles are intrinsically more or less authoritative than a plain Google Profile.

• Google says that as an alternative to linking, authors may publish, for all to see, a verified email address in their Google Profile and ideally include the same email address in their articles.

It it isn’t so clear how Google could reliably match the two if the email address isn’t included in the website containing the author’s content – anyone who has experienced Google Maps local business data merging knows what can happen when the algorithms get it wrong.

A second problem with this suggested implementation is that people should never expose an email address in plain text on the Web unless they want to radically increase the number of fake Rolex watch offers, Nigerian 419 solicitations or worse that arrive in their email inbox. Email spammers “harvest” bare email addresses which appear on the web for these nefarious purposes. While some email systems (read: Gmail) have great spam detection, best practice says to avoid the problem at the source.

In its rush to promote Google+, Google is aiding and abetting email spammers. Note that the email verification process just proves the email address works – there’s nothing in the verification that says the person is who they say they are.  This is not the same level of Google Profile verification Google once offered as part of their Knol system.

• The “enhanced” Google+ Profile does contain additional signal information which Google might argue is important in determining the authority of a specific author. It tells Google how many people have the author in a circle, how many people have shared a contribution from the author in Google+ etc. Yet Twitter profiles, as an example, contain essentially the same information.

• At the end of October 2011 Google added Google+ specific annotation to authorship rich snippets – the related announcement took for granted that authors use Google+ profiles; there was no mention of basic Google Profiles.

• In November 2011, Google began to prepend a + to an author’s name in the SERP byline. Google has promoted this syntax in Google+ as an alternative to the de facto standard @ notation first popularized on Twitter to refer to a specific person.

What Say Bing, Yahoo, Yandex & Blekko?

Bing supports some rich snippet formats, at least for users in the US, specified using microformats or Schema.org markup but there’s no sign they’ve yet implemented author specific markup, nor has search alliance partner Yahoo.

Yandex, the leading search engine in Russia and some eastern European countries, is beginning to support enhanced search result mark-up as a member of schema.org group. At emerging search engine Blekko, folks are aware of microformat markup but they don’t seem to be using it themselves.

What Google Should Do Next With Author Markup

• Document how to implement search engine agnostic authorship markup using the Schema.org specification and demonstrate that Google does indeed support it.

• Provide greater clarity as to who is eligible for authorship enriched search abstracts. A multitude of Google blog posts encourage content authors to deploy authorship markup without setting reasonable expectations as to when, if ever, an author would actually be eligible for authorship snippets. Add “strength” feedback to the snippets markup testing tool. If results are limited to English searches on Google.com as seems to be the case, say so. The casual international reader of Google announcements isn’t going to be aware of this important detail.

• In a similar vein, provide guidance on what authors should do to avoid losing rich snippets, e.g. will changing to schema.org markup penalize an author?