Why is a Robots.txt File Used?

[Arunsingh]

Why should I have a robots.txt file?

Thanks...

[Rajdeep Bose]

This section has a few handy examples.

To prevent FreeFind from indexing your site at all:

user-agent: FreeFind
disallow: /

To prevent FreeFind from indexing common Front Page image map junk:

user-agent: FreeFind
disallow: /_vti_bin/shtml.exe/

To prevent FreeFind from indexing a test directory and a private file:

user-agent: FreeFind
disallow: /test/
disallow: private.html

[blessy_smith]

Robots.txt are also helpful in blocking the bots from indexing directories that contain scripts. If you have a very plain site that you would be ok with having the engines crawl everything than you do not need a robots.txt.

[brandappz]

Robots.txt is a simple text (not html) file you put on your website root directory to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all.

[madtteam28]

Robots.txt is a text file that consist of half of the URLs of some pages that you don't want search engines to be crawled.

[Johnpaul]

If you dont want to crawl the thing in search engine, just create robot.txt and add them in it. Search engine will nevrer crawl them.

[Mariajoseph]

To tell which search engine and which pages to crawl and index.....

[godwin]

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[dombowkett]

Robots.txt is used to check which portion of website we have to index OR not.

[Maria Jonas]

Why should I have a robots.txt file?

Thanks...

Have you Googled this? If not, do it now!

[GeethaN]

A robots. txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.