Google, Yahoo, Bing, Baidu, Yandex all have their own way of crawling web pages. But I should say that Yahoo and Bing are very big misers.
Google crawls a site with the speed corresponding to the frequency and quantity of updates of that site, till today Googlebot is the best crawler there has been (among search engines), they will add your new site to their index in no time if it's worth adding and displaying in search results, though they are the best but they are still slow (it is to me, will be to you too if you have a bigger site running overnight).Though google considers link popularity in ranking webpages for search results, it almost doesn't consider link popularity of a site for crawling it
Baidu and Yandex visit sites periodically, not sure if they consider the updating speed of site or the link popularity.
If I have to mention a single reason why Yahoo and Bing have lost their war against Google, it's really really because of their incapable crawlers. Both of them consider link popularity, domain age, domain authority and of course the size of the site for crawling it, it makes them slower and because of the millions of new webpages showing up everyday they will always be behind Google in time.
Yahoo is considerably better than Bing in Crawling, Yahoo has crawled one of my site (which doesn't have external backlinks) almost completely in one year, bing has listed only the main page yet
- Permalink
- The Analyst
- 25 Feb 2010 12:56 AM
- Comments (0)