Introduction to Robots.txt

[87nitingarg]

The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.

[smith29]

Robot.txt is a text file... Thanks for sharing..

[rajjan011]

robots.txt file is very important for crawl. it's really very simple to create this files.
Dhanaulti Camp (http://www.arounddelhi.net/?page_id=7093)

[thomson]

What's its usefulness for a website ?

[sabrinasai]

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

[watson123]

Thanks for sharing the information

[samdorjey]

Robots.txt is a text file you put on your root directory. The location of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it. It is used to tell search robots which pages you would like them not to visit.