denniemark
04-19-2012, 05:46 AM
I hope, you all must listen about Crawl delay. Lots of webmaster use “crawl-delay” word in the robots.txt file. I searched about this, on Google and got lots answer. But, I did not get any answer, which satisfied to me. So, I made topics on this and posted here.
Sometimes, you have seen this Line in robots.txt file.
User-agent: Googlebot
Crawl-delay: 10
Can someone explain? What is the meaning above line?
Sometimes, you have seen this Line in robots.txt file.
User-agent: Googlebot
Crawl-delay: 10
Can someone explain? What is the meaning above line?