PDA

View Full Version : Crawl-Delay – What,s the meaning of this?



denniemark
04-19-2012, 05:46 AM
I hope, you all must listen about Crawl delay. Lots of webmaster use “crawl-delay” word in the robots.txt file. I searched about this, on Google and got lots answer. But, I did not get any answer, which satisfied to me. So, I made topics on this and posted here.

Sometimes, you have seen this Line in robots.txt file.
User-agent: Googlebot
Crawl-delay: 10

Can someone explain? What is the meaning above line?

watson123
04-19-2012, 06:48 AM
Though it is not "a standard" within the "non standard" robots.txt, many bots obey the "Crawl-delay:" parameter. Since drupal sites seem to be popular with search engines and lost of people have more aggresive bots than visitors at their site, it might be wise to slow down the robots by adding a robots.txt line lik