What is robots.txt?

[pranav]

What is robots.txt?

[evabrown]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sensitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[PoolMaster]

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. The slash after “Disallow” tells the robot to not visit any pages on the site.

[seocontentdada]

Robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots which pages on your site to crawl. It also tells web robots which pages not to crawl. The slash after “Disallow” tells the robot to not visit any pages on the site.

[albertocosta]

The robots.txt is a text file using a precise format that allows a Webmaster to control which areas of his site an indexing robot is allowed to scan. This text file will be available at a specific URL for a given site.