By default Google sets your site's crawl rate according to the frequency of new pages that you create, you can change the speed up to a maximum of 0.5 requests per second (i.e 2 seconds delay between requests) inside webmaster tools at Dashboard >> Site configuration >> settings, as your site grows you can increase the crawl rates beyond the previously maximum 0.5 requests per second.
Your site's crawl rate doesn't randomly increase, it increases in levels
#1 Maximum of 0.5 requests per second
#2 0.75 requests per second
#3 1 request per second and more
Your site's crawling speed may jump to any of these levels depending on the speed of growth of your site, please mention your site's crawling speed if it's more than 1 request per second
- Permalink
- The Analyst
- 17 Feb 2010 6:34 AM
- Comments (1)
April 23rd, 2010 at 1:39 pm
Im surprised to have found this blog, I though I was the only one writing about this…thanks.