Robots txt

[ryansmith]

What is Robots txt?

If you have any suggestion, please suggest me.

[dennis123]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[BenWood]

It is used for security purpose of webpages.

[edwina]

Robot.txt:
It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

[Piers Rollins]

Actually, Robots.txt is the text document file, which are mainly used for the purpose of showing/avoiding, wanted/unwanted pages to search engines and users

[ajay49560]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[angelinagertz]

Robots.txt is used to tell the Google spider to which page is crawled or not by allowing and disallowing.

[BerthaJones]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[irfan]

A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.

[ryansmith]

I appreciate your answer.

[jaysh4922]

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

[rajanvefgus]

Robots.txt is a file which is helpful to crawls and dis crawls the site pages. using
User-Agent: *
Disallow:

picnic spot in delhi (http://www.arounddelhi.net/?page_id=203) | one day outing in delhi ncr (http://www.conventionindia.in/Daysout.aspx)

[ramskl]

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

[bestpolesaw]

Nice information thanks for sharing.

[Sandhiya]

Its a google spider which not allows the google to crawl the pages/websites.

[CarolineMurphy]

Robot.txt is text file which you put on your website to tell search engines which pages you would like them not to crawl. The file looks like below

User Agent: *
Allow: /
Disallow: /admin

[bestpolesaw]

robots.txt file create by webmaster for indexing website fast.

[raheel]

Nice information thanks for sharing.

[ArkPresentation]

Robot.txt: use for allow and disallow on website.