Robot File..?

[hemantets]

Hello Friends What is Robot.Txt...?

[timmoores]

Robots.txt file is a file, which is initially checked by the search engine crawlers (robots), and based on it search engine decides, which pages have to do index, and which pages do not have to index. If you want search engines does not access the certain folders, you can use simple robot txt command "Disallow: /cgi-bin/" (without the quotes) – and the directory will not be accessible for search engine. Some search engine optimization experts claim that bots do not follow the rules, but it is not true. You can also use robots.txt file declaring your site map without creating a Google and Yahoo account, in which you have to submit it manually.

[john515]

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not.

[jasoncamroon]

A robots.txt file is also called the Robot Exclusion Standard, Its one type of search engine program to pass the request to search engine robot to ignore specific directories or files when search engine robots(spiders) crawl a website.

[giannabosanquet]

Well described in above posts. You can create a robots.txt file from free creators or through webmaster tools.

And manage which pages you need to get crawl or which to not or for any particular bot or crawler.

[markdev]

Robots text file is the file where you can allow or disallow user agent means robots or spiders to crawl or not to crawl that web page which you restricted. It is not necessary but good to have.

[kewin12345]

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty.