Creating An Enhanced Listing With Yahoo!’s SearchMonkey Application Development Platform

I’ve been playing around with Yahoo!’s slick new SearchMonkey Application Platform which was first introduced by Amit Kumar, Director of Product Management at Yahoo! Search, at the SMX West conference earlier this spring. Yahoo! hosted a Developer Launch Party yesterday in Sunnyvale to open the platform up to all developers. I created an example application for the Search Engine Land site, so read on for a description of the platform.

SearchMonkey allows publishers to develop a custom layout for their pages’ search result listings — instead of just a title hyperlinked to your page along with some description text snippet, you could include other components, such as a small picture icon, related links, rating stars, and brief lists of specific info about the page’s subject matter. Note that your users have to add your custom layout to see this in their search results.

As I took a look at the application development interfaces, I was pleased to see how Yahoo! had set the whole thing up. There are two main parts of the platform:

• Data Services through which you can use either the elements that Yahoo! already collects from your webpages or set up your own custom data services to provide the application with structured data.
• Presentation Applications, which allow you to take the data, format it, and deliver it up through the listing area.

Sites that already use some semantic markup such as Microformats will be a bit ahead of the game for developing applications through SearchMonkey, because such markup can really help in setting up the data extraction for special presentation applications. But even if your pages don’t use semantic markup already or you have difficulty in setting that up, SearchMonkey has sufficient flexibility to enable you to show the system how to extract data from your page by setting up an XSLT file that specifies what information to extract from your webpages and what variable names to assign that data to.

Setting Up an XSLT File

The XSLT lets you “tell” the application the locations within the document for finding specific information — it’s sort of like a map for screen-scraping a document. For this example, I set up an application that would allow me to display a small thumbnail image beside each Search Engine Land (“SEL”) listing, if an article contained an image illustration. I was able to do this by providing an instruction that called for returning all images found within the DIV tag on SEL pages that had an ID value of “maincontent” — this DIV tag was a convenient container definition I found that wrapped up all the body of articles on the site. I set the XSLT to iterate over and return all images because I found that many of the articles re-used the same column icons over and over — such as the “Locals Only” icon or the “100% Organic” icon.

While it might’ve been tempting to use those column icons to decorate article page listings in Yahoo’s search results, their best practices recommend against using placeholder icons or generic images over and over because they provide little value to end users, and end users tend to blank out things that may appear to be advertisements. So, I returned all the images within the main content since some articles had unique illustrations, some had column icons, and some had neither.

I also got the XSLT to return everything wrapped in an H3 tag with a class value of “date.” This field on the SEL site has a time/date stamp along with authors’ names — I thought it would be great to harvest out each article’s author name and display that along with the listings.

Creating the Application

After setting up the Data Service for SEL, I then clicked to create an application. When setting one up, you create a small profile entry with the application’s name, type of result presentation (Enhanced Result or Infobar below a listing), description, specify a category (I used “News & Blogs” for SEL), and upload a small custom icon for your site.

Next, you set up a trigger URL pattern for the types of pages that you want to create a treatment for. In this case, I set up a pattern that includes all SEL pages, but if there had been a subdirectory containing all the articles, I would have set up a matchstring pattern for only that directory. I also entered a handful of test URLs for previewing the treatment I was designing:

After this I specified which data services the application would use, and then went into the appearance specification to map the data into the presentation layer.

The appearance interface is probably the coolest part of SearchMonkey, IMHO. It allows you to write PHP code that executes within Yahoo’s search results presentation layer in order to display your custom listing treatment. Here’s the code I wrote for the SEL application:

I kept Yahoo’s default title and description snippet (“summary”) for listings, since those already seemed pretty good for article pages. For the images, you can see I pull in the first two pictures that may appear within articles, and then I run a conditional check to see if the image URL contains “100.jpg” or “button,” and I don’t use the first pic if it does. This eliminated all the standard column icons and social sharing buttons that appear at the ends of articles. If the first pic on a page doesn’t appear to be an article illustration, I check to see if there’s a second and use it, also checking that to eliminate spacer gifs and such.

Finally, I take the text from the dateline of the document, split it at the word “by,” and use the text afterwards as the author’s name.

Display in Search Results

Here are a few different example listing displays that are generated from this app:

As you can see, the addition of the thumbnail pics often make the entries look a bit more engaging. Many articles do not have illustrations or photos to go with them, and those entries still appear very similar to standard search results listings.

Things To Watch Out For

The trickiest part of developing the applications appears to be the need for developers to keep in mind all of the various exceptions to page code formatting which can occur across large sites, particularly in sites with many types of pages. Developers need to code in sufficient conditional checks to ensure that scraping code for one type of page doesn’t display incorrectly for other types of pages, and they need to set these conditional checks up in an efficient manner, or else the processing of the application will take too long (Yahoo! has a processing time limit, and will revert to standard listing layout if an application hits that time limit — this is also a major factor for sites that are very slow at returning resource requests).

One item I considered adding to the SEL application was Sphinn rating numbers for those articles which had been Sphunn. This introduces extra complexity, however, since Sphinn ratings are delivered up from a different domain than SearchEngineLand.com — Sphinn ratings that you may see on article pages at SEL are served up dynamically from Javascript calls to Sphinn.com, so the actual rating value for any given article isn’t resident within the HTML code of the article’s page.

SearchMonkey does have a method for allowing this — they have a Web Service data service with which one could retrieve the off-domain sources of info. Unfortunately, that off-domain type of application can only be used with an Infobar implementation, rather than the Enhanced Result listing that I wanted to use. I believe the Enhanced Result is much more engaging than the little infobars (which will dynamically execute below search result listings when clicked, expanding and displaying other types of content).

Another solution to this would be to customize the programming that delivers up SEL article pages — enabling the Sphinn ratings to be dynamically queried by the SEL servers when an article page is requested, and then parsing the rating display directly into the HTML of the pages. If this was done, I’d then be able to scrape the rating value off any article page, and display it in the special presentation application for Enhanced Results.

This particular issue is likely to impact many different publishers who desire to integrate with Yahoo’s SearchMonkey, since bloggers might want to display their Digg ratings with postings, and many other user ratings/reviews services are implemented through similar iframes or javascripts, such as BazaarVoice, which is used by a great many internet retailer sites.

How SeachMonkey Applications May Evolve

Now, one question I had in mind when SearchMonkey was first announced was something like, “If all listings had this jazzed-up treatment in Yahoo, won’t it end up making search results awfully visually busy?” While having a few jazzy listings might be cool, I could easily see the usability and user experience degrade if their search results became too loaded up with stuff. It appears to me that Yahoo is intending to be cautious about this as well, and to handle this I believe they’re carefully assessing which presentation applications they may allow to be default for the public. An application would likely need to be useful to the public in order to get promoted into standard availability for all Yahoo! users. Otherwise, these presentation applications will go into a gallery where Yahoo! users may select ones that they’re interested in, and it will be a part of personalization options. You can also promote your presentation application on your site.

The development tools allow anyone to build presentation applications involving any website — which is very flexible (if not downright savvy of Yahoo!, since this could definitely allow application treatments to become expanded very rapidly). It appears that any application that’s to be considered for being automatically available to all public users would likely have to be created by a user who has authorization for that domain through Yahoo’s Site Explorer. Otherwise, I think you could develop apps just to be included in the Gallery for users to optionally select.

Some companies appear to be planning to use the SearchMonkey tools to develop presentation applications that would particularly benefit their employees — perhaps “bubbling up” information from webpages to be displayed directly in search results, reducing the need to click through to webpages. This could also be very useful if Yahoo! Search is powering custom site searches for companies or educational institutions.

SearchMonkey Developer Challenge

Yahoo! has also launched a SearchMonkey Developer Challenge, and developers who create applications that they particularly like could win monetary prizes, with the “top banana” winning a $10,000 grand prize. I suspect that the applications which are particularly useful to the general public and which are engagingly developed will be the most likely candidates for the prizes.

The concept of allowing webmasters to have a say in how their search listings may appear is extremely strong. Yahoo!’s development of SearchMonkey proves that they can come up with stuff that’s very innovative, even from their underdog position in the search world.

It’s going to be very interesting to see which SearchMonkey applications are released into the wild in the coming weeks and months, and how those innovations impact sites’ traffic. For some types of sites and presentation treatments, the special listings will likely attract more clickthroughs than previous standard treatment. Yet, if a site like an internet yellow pages has their listings displaying a business’s address and phone number, it could actually reduce clickthroughs since users may already get everything they desired from the SERP listing. Hopefully someone like Enquiro will conduct an eye-tracking study and see how these new listing treatments affect user browsing and click patterns.