Googlebot now crawls the web from Korea and Russia too

Last month, at the International Search Summit conference in London, a Googler Gary Illyes, who was presenting the controversial topic of website geotargetting and Hreflang, revealed an interesting piece of information: Googlebot is now crawling the web out of three different geographical locations:

• Mountain view:  Most SEOs know  that this is the ‘defacto’ starting point for the Googlebot to start crawling the web
• Russia: although he did not specify where in Russia, I am not surprised about this as Google has been struggling to increase market share in this region for some time due to  stiff competition from a stronger local opponent.
• SouthKorea: not a surprise either.  Their two main local search engines Naver (72%) and Daum(18%) account for 90% of all searches in the Korean search engine market.  What could Google possibly want here?

So Why SouthKorea and Russia?

Gary mentioned that they were not keen to let any more information go around as to the reasons why Google may now be crawling the web out of those two new geographical locations. So with this post, I am just throwing my  two cents on this matter.

Google is clearly not hiding his plans as its engineers go out to the conferences and use this information on their decks. But what are the reasons and Google’s interest in throwing their crawler from those two additional geographical locations?

It’s very clear:

Korea, next to Russia and together with China form the three main and major Google competitors in the international search engine market land. Google fears that some of these search engines will at some point defeat its almighty presence in other territories.

Google wants to dominate in every single market. Google is trying to avoid big competitors like Baidu, Yandex and Naver to become bigger and gain more market share. The three main markets where they don’t have the lionshare’s chunk of search are China, Russia and Korea.

We also have Japan and Taiwan where Yahoo boast over 50% of search share but their results are served up by Google, at least in Japan. So effectively Google controls and owns most of the search results in those two countries too. This renders us with just three: China, Russia and Korea as the main difficult markets for Google to penetrate and dominate.

So my take is that Google has brought the crawler here in order to begin drilling deeper into the search landscape in these markets.

 

source: Preston Carey’s slideshare

We are going to leave China out of the equation as Google is being restricted by China’s censorship internal wall and not as easy market to penetrate into. Years ago they abandoned their attempts to walk inside the Chinese wall. So leaving China aside the two other ones on the spotlight are Russia and Korea.

Google wants to learn and understand more about those two local market search engines, their linguistic nuances and about the search results that make people want to use their local versions as opposed to Google’s own engine. Google wants to be able to provide an equal, similar or better search engine results than those offered by Korean’s Naver and Russian’s Yandex.

UnGagged, the SEO UnConvention

 

Some Stats about South Korea and Russia

Google boasts a very humble piece of the south Korea search engine market share (5%), with Naver having the lionshare of search.  The leading search engine in this market in Naver, with a difference, followed by Daum, and then Google in the last position. This is, potentially, the most likely reason for Google to take base on their crawler and strengthen their presence their by opening new offices. Otherwise, why not Japan which is also quite a strong economy and with a booming internet market?

In Russia. Google has got a larger marketshare than in Korea: about 25% . Yandex obviously enjoying the larger chunk: 65%. What is bad news for Google is that Yandex has been making plans to expand in recent years and has already began expanding onto emerging markets in LatinAmerica, Europe, Turkey. Yandex is nowadays the fastest growing search engine.

So as Yandex enjoyes 62% of share, Google may want to get better and more insightful ‘intel’ about the Russian market by establishing base there.

Sure, but what about Baidu, they are also expanding and penetrating into other markets?

Yes, but as mentioned before, as they are not allowed to set up data centres in China unless the comply with censorship, which currently they dont, they are kind of leaving China and Baidu aside for the time being. However, my opinion is that Google will, in the end, bite the bullet and comply with Chinese censorship and government control rules on Internet use.

I wouldnt be surprised if within a year or two, Google takes a completely different steer and ses up base in Beijing or other key location in China.  They could soon start crawling the China’s wide web at ease and gathering data, insights and signals that helps them streamline their results in Chinese in order to compete better with Baidu and stand a greater chance to increase market share in this huge market.

 

And what about Bing and Yahoo?

Of course Bing and Yahoo are competitors too, but they are kind of US companies, therefore it is easier for Google to get key info about how they operate. Plus they are search engines that also struggle with international penetration in certain markets.

Yahoo for example is big in Japan, and in Taiwan and Hong Kong, but they don’t present a massive problem to Google’s monopolistic ambitions.

Yahoo’s UVP in Japan for example is not just ‘search’ but a range of localized mobile and social services that they provide to the local markets, which makes them unique and distinguishable in those countries. But are they truly competitors that may present a block to Google worth worrying about. I’d agree that Yahoo and Bing may be serious competitors to Google but on a different level.

What do you think? Are you able to speculate the reasons why Google may now be crawling the web out of South Korea and Russia and not anywhere, like Spain,  Malaysia or Wallis & Futuna?

 

Articles I have read as reference for this post :

 

http://googlewebmastercentral.blogspot.fr/2006/09/how-to-verify-googlebot.html

http://www.link-assistant.com/blog/google-vs-naver-why-cant-google-dominate-search-in-korea/

http://returnonnow.com/internet-marketing-resources/2013-search-engine-market-share-by-country/

http://searchengineland.com/matt-cutts-in-south-korea-109861