What is the robots.txt file?

[bhadriram]

Robots.txt file which tell to google to crawl the webpages or not

[webcreations]

A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.Robots.txt is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

[alaxhooper]

Robots.txt is a file placed in the root of a domain that is used to inform search robots about the structure of your website. This is often used to block robots from specific folders or pages of your site.

[yepionline]

I really like this post, very informative and very useful

[Vamp1re]

Robots text file are those which forced to search engine how to crawl and index the your site or web pages.
It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

[rsewak]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

[blessy_smith]

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items.

[Markgibson431]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[saiyadgulbaz]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[Manov]

The below link gives you an detailed information about Robots.txt
http://www.javascriptkit.com/howto/robots.shtml

[barbu5]

When you add your website in google.com/webmasters it generates itself the robot.txt. you don't need to add any file or change anything. If you want not to index meta, category, archive download "all in one seo pack" plugin and check not to index them.

[Jitendra Shukla]

robot.txt is is a file to prevent Google's Crawler to read any particular page. it is basically use to hide pages which we don't want to index by Google.

[SANTOSH123]

Robots.txt a text file made on notepad which instructs search engines about their allowed crawling or visitng areas on full website.
Robots.txt file is used to what pages you want crawler to crawl or not.

[Jitendra Shukla]

Most of the e-commerce site use robots.txt to hide few important pages like Payment and Thanks us page.

[sptechnolab]

robot.txt file is used to direct or to tell web bots which pages and directories to index or not to. This file must be compulsory placed in the root directory.