September 26th, 2011

Yandex – Yet ANother inDEX?

There are parts of the world where Google is not dominant. In Russia the search engine Yandex is the market leader with a market share of over 60%. Helped by rapidly growing internet usage in Russia, the search engine doubled the number of searches it handles between 2008 and 2009. For online businesses operating in Russia there is no question about the importance of Yandex (for more about search engine marketing in Russia read our Internation Search Review post) but since the launch of an English language search engine in 2010 should the rest of the world be thinking about Yandex?

There are two aspects to this question:
1. Will Yandex gain a large worldwide market share?

2. Are there other reasons to observe what Yandex is doing?

Will Yandex gain a large worldwide market share?
In my opinion, no. Right now people have no reason to use Yandex. It is not integrated with any of the online services commonly used in the West, nor is it the default search engine on any of the main browsers. The only ways Yandex can increase market share are either by spending a lot of money on advertising (this is working, but very slowly, for Bing) or by being better at search than Google. Unfortunately for Yandex, they can’t just be a little bit better they need to be a lot better; studies (by Microsoft) show that people say the quality of results from Bing are equal to those of Google, but only when the Bing results are wrapped in Google branding. Any new search engine that wants to dominate the market needs to be an order of magnitude better, just as Google was in 1998.

The search technology behind Yandex
Google beat the competition with their PageRank algorithm. Page and Brin realised that strong webpages were more likely to be linked to from other strong webpages. In other words, they picked a feature that they thought good webpages should have and then built their search engine to rank pages with this feature.

Yandex’s MatrixNet algorithm is very different; given a list of good pages for a queryspace, MatrixNet uses machine learning to decide which features distinguish them from the average. Then they rank pages with similar features in that queryspace. This method is a great defence against spammers because any feature that becomes common is no longer a powerful ranking signal. For example, if everyone has an optimised title tag then having an optimised title tag is not a signal of quality.

The main weakness with the MatrixNet approach is getting the list of good pages to begin with. The internet is too large for this to be manually curated so there has to be another algorithm to generate the list of quality sites. This algorithm must be very conservative in the sites it selects, otherwise results quality will suffer a lot; imagine if having a large number of AdSense ads became a positive ranking factor!

Google’s recent Panda updates use a similar approach. Matt Cutts (Head of Web Spam at Google) has said that they “came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side”. However, this algorithm update can only reduce rankings, not increase them so Google do not need to be as conservative with how it is applied (some site owners say they should have been a lot more careful).

As evidenced by their Panda update (and many other projects), Google has the technical ability to do machine learning at web scale. Should Yandex’s approach begin producing SERPs of amazing quality then Google can copy their approach before Yandex’s market share reaches critical mass. This is why Yandex need an order of magnitude improvement over Google; they need to capture a large amount of market share before Google improve their algorithm to match.

Why you should pay attention to Yandex
Like Yandex, the browser Opera also has a large market share in Russia without being a big player in the West. Opera introduced features like tabbed browsing and “speed dial” that have since been imitated by Firefox, Chrome and others. Web designers watch how Opera are innovating because some new features will cross over into the mainstream.

Similarly, you should keep an eye on what Yandex are doing because they take a different approach to search and successful features from their algorithm are likely to appear in other places.

Blog post by Richard Fergie, Consultant at Reform