Yandex: Not Copying But Searching For Google’s Underbelly

Ilya Segalovich, the chief technical brain behind Yandex, is a very nice guy. I’m not just saying that because he allowed me to interupt his busy schedule, but because you can’t have a conversation with him without thinking that you are both sharing and learning at one and the same time – he listens intently to what you have to say, expands it and develops it and gives you more back each time.

The elephant in the room during our conversation was Google. You can’t discuss Yandex without making comparisons to Google – as a sort of yardstick.

But that doesn’t mean that Yandex copies Google, as Ilya is very keen to point out. The first slides he has lined up to show me, were originally put together as a retort to Robert Scoble who described Yandex as a clone of Google in answer to a question on Quora.

That’s not going to go down well at Yandex towers in Leo Tolstoy street where everything is developed “To answer all the questions of the user” and definitely not because a Californian competitor – who’s also active on Russian turf – happens to come to similar but not identical conclusions.

The Yandex – Google Timeline

In fact, the slides list a range of things which Yandex launched first pointing out firstly that they in fact launched as a search engine in 1997, a year earlier than Google. Yandex also launched maps first in 2004, Google a year later in 2005. Yandex was the first to launch news search in 2000, Google in 2002. Blog search came out of Russia in 2004, but out of California only in 2006. Yandex had already launched an RSS aggregator in 2005, Google followed in 2006.

For the sceptics amongst you, here’s more proof you can check yourselves. According to Whois records, google.com was registered on the 15th September 1997 and yandex.ru just eight days later on the 23rd September 1997.

Meanwhile, yandex.com was registered one year later than its Russian forebear whereas Google took another six and half years to claim google.ru on the 4th March, 2004. So if Yandex copied Google after eight days, well you at least have to credit them with amazing foresight!

The Birth Of Yandex

Ilya also shows me this fascinating picture which was taken in 1981 when, as young men, Arkady Volozh (on the far left) and Ilya (center) were just going up to university and yet already knew each other well. It was to be another 16 years later in 1997 that yandex.ru was finally launched and a great many adventures inbetween.

Ilya’s father was a well known Russian geologist credited with identifying tectonic anomalies in the Urals resulting in the discovery of huge reserves of Chromite. He was awarded a State prize in the days when the Soviet Union did that sort of thing.

Arkady Volozh is also the son of geologists and the two originally met in Almata in Kazakhstan in their formative school years. (Curiously, Arkady Volozh also mentioned to me separately that the families of Sergey Brin and Arkady know each other too).

Ilya left university keen to develop software but in 1989 he thought that “Search was a very stupid idea” – to be fair there was no Internet and no Web at that time.

Computers were barely computers (I can vouch for this – I remember the steam driven machines of that era) and to load the early data and text search software Comptek, for instance, meant loading 9 floppy disks of installation code. Ilya had his own software plans and team.

Finally, in 1990, Arkady had his way and managed to recruit Ilya to the Arcadia team. Ultimately, this led to Ilya having the chance to make his mark on history since a few years later, Russian venture capitalists were looking to invest in anything to do with the Internet.

This brought focus onto a particular Arcadia project which as a result needed a name. In 1993 Arkady Volozh and Ilya Segalovich developed a search engine for “non-structured information with Russian morphology”.

Ilya proposed the name “Yandex”, derived from the idea of “Yet Another inDEX” though in Russian there is an extra play on words as the first letter “я” (pronounced “ya”) also means “I”. It stuck and in 1997 yandex.ru was born, the search engine launched in 1998 and in 2000 Yandex became a company.

Back in 1994, the kind of search Ilya was working on is illustrated by the image below which illustrates the key feature which made Yandex great and which Google didn’t replicate until 2006, some 12 years later — that is Russian morphology text search.

The search term in the search box is a two word search term in one particular grammatical form – the highlighted red “finds” in the text show terms which, though they match the search term, have been changed in spelling by the rules of Russian.

The graphic also illustrates quite clearly the kind of work the Arcadia/Comptek team were working on and which later evolved into web search.

In fact, the parallel paths of development of the Google elephant and Yandex are fascinating and dominate a lot of our conversation. Ilya refers to a statement published on Google explaining its search techniques in 2000 which said that, ” Google does not use “stemming” or support “wildcard” searches. In other words, Google searches for exactly the words that you enter in the search box.”

The wording implied that this would result in more relevant results. Bearing in mind that Yandex’s birth in 1997 had directly followed a solution to conduct “inexact” search using stemming which was the only way relevance could be achieved in Russian, you can understand that there must have been a few wry smiles in Leo Tolstoy Street.

I ask Ilya when he first became aware of Google? “Danny Sullivan brought it to my attention in an article”. I believe he means a piece called “Counting clicks and looking at links” from August 1998 announcing the new experimental Google still on a Stanford University URL.

Ilya goes on to describe the history of Google’s roll-out to Russia, “In 2000, Google was focusing on distribution deals, 2001 was the year of internationalization where they were focusing on the completeness of the index and had Russian content, but it was not until 2006 that we considered Google a threat. That was the year Google started supporting Russian morphology”.

He observes, “We were lacking resources and didn’t have the right people in the right jobs. We had to make changes and to re-organise. Fortunately, we found some great guys who saved us.”

Intriguingly, Ilya attributes the Yandex versus Google performance in 2008-2009, when Yandex lost market share to their search performance in 2006-2007.

“There seems to be a two year delay before users really notice the difference when you’ve made improvements,” he says. “We introduced MatrixNet machine learning in the beginning of 2009 and only really noticed a significant difference in search shares in 2010.” He adds, “It’s harder to develop a search engine than a space program, five countries have their own search engines, 20 have space programs.” At this point, Head of Search, Anatoly Orlov has joined us and chips in, “Search is rocket science!”

How Yandex Is Different From Other Search Engines

The Yandex search page is quite different from Google’s. Whilst Yandex has a search box only page for those in a hurry on the short URL, Ya.ru, the main page looks rather more portal-esque. Yandex did test using the search box only page for a few weeks, but found that the majority of Russian users prefer their normal busier page.

Ilya points out that people can change which search page is the default, “We even allow users to turn off some of the ads,” he says. One thing is very clear, Yandex is and has always been obsessed with search and with satisfying the needs of the user.

The search home features no entertainment, no dating and Ilya confesses, “I don’t watch television, but I understand people do watch it.” There are apparently a lot of children at home so that’s perhaps understandable. The normal search page may appear like a portal but there’s more to it than that.

Yandex views it (explained below) as the answer to the question, “What’s new? What’s new around me?” So that’s why there’s news, weather, traffic information (crucial to Muscovites), market news, trending money and automotive — Yandex’s top vertical market. This being Russia, the price of oil is included in the list of exchange rates!

Like all search engines, Yandex watches its search quality performance very closely. Ilya shows me a series of graphs from AnalyzeThis.ru – a Russian independent SEO firm which tracks the quality performance of search engines in Russia. Ilya was keen to stress that this is just one of the sources that provides such data — but I found the result interesting and worth sharing.

The first graphic below shows the percentage of missing navigational results across Yandex, Google and Bing in Russia where the lower the score the better the result.

Despite a brief Yandex blip (the reasons for which are unclear), overall Google and Yandex are tracking each other on this metric very closely giving a very similar quality of result to the user – but the Bing results are not so good, although they have shown some improvement since October last year.

The next quality measure relates to the measuring whether original text is ranked higher or lower than content copies – with a higher score giving a better result. This is a major problem for a lot of us who want to allow syndication of our content but don’t want the syndicators to achieve better results than we do.

In Russia, Google is the poorest performer on this measure beaten even by Bing. Yandex scores much better than either Google or Bing with a score of 50% over Google’s 20% and dropping performance. Of course, the Farmer update won’t be counted in these figures but if you were Google looking at these figures – you’d have wanted to target content farms too!

We now turn to a favourite topic of everyone – spammy results. The better results for users are those with a lower percentage of spam – naturally.

Guess who wins? In the AnalyzeThis.ru index since 2009 to the present day, Yandex has never been beaten by either Google or Bing. They have won a reputation for being very tough on spam indeed and these figures seem to confirm that.

Does a search engine provide purely commercial answers to commercial questions? Here, the measure in our chart below is better, the lower the percentage and once again Yandex wins in Russia beating Bing and wacking Google.

Most significantly, Yandex is showing a continuous downward trend, Bing similarly and at points has been in the lead. Google has not been performing well on this measure. Perhaps Google doesn’t target this measure?

Yandex may be perceived to be serving only one country – but that’s not quite true. There are 15 countries which use the Cyrillic alphabet and 77 regions in Russia itself – and Yandex aims to be local everywhere. Culture, standard of living and average income are very different across the wide area Yandex serves – so search results need to vary dramatically too – including autocomplete.

Our final chart looks at the percentage of local results with a higher score the winner. Here, Yandex seriously kicks butt with Google way behind and as far as Bing is concerned, well their local results look like they might be an accident.

Yandex.com, the Latin-alphabet and English version of Yandex is described as “experimental” and, despite creating a stir with bloggers in the west, it is explained that Yandex had been working on indexing sites in other markets for some time and the intention was not to go international, but to enable Russian users to carry out Latin text searches without leaving Yandex.

Roughly 7% of Russians know English, but only 1% know English well- enough to be happy to navigate in English and to English-language sites. Nonetheless, the Russian press got very excited about Yandex heading into international markets.

So I ask Ilya directly if there are any plans to expand Yandex internationally. The answer confirms that there are plans, but of course nothing specific they can share with me.

Then, we get into what you need to succeed with a search engine in local markets. The language and handling the language well is a major factor but, “Having a local database affects some 18% of searches.” There are also market specific things you need to do. Says Ilya, “We’re getting pretty clear to understanding how to do it!”

Of vKontakte there is a feeling that not enough is being done by their parallel in the social network world – in other words, the local player and number one social network. It turns out though, that “vKontakte” is the number one search term on Yandex reflecting the Facebook position in Google’s logs.

I conclude that Yandex is watching Google very closely and looking for its weak underbelly before launching a carefully thought out attack. It seems that this will be in markets where Google is number one, but only because there’s no contest and in fact Google doesn’t really have a solid infrastructure and localized approach in that market despite its position – and no local database.

We’ll be dealing with Yandex’s machine learning approach to its algorithms next time and the implications for Russian SEO and SEO in general.