Why we use Robots.txt File ?

[gaurav467]

anyone share your information?????

[general]

robots.txt are also helpful in blocking the bots from indexing directories that contain scripts. If you have a very plain site that you would be ok with having the engines crawl everything than you do not need a robots.txt.

as for xml sitemaps, just because the engines say they would like you to do something doesn't mean you should do it. for good webmasters i see no benefit from xml sitemaps. but we should discuss that in a seperate thread ;)

admin pages , paypal pages and other pages we can block here not to crawl these links.....

[yourmobisite]

Hello..
Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders.it is an important factor for the webpages being indexed properly and very easy to setup.

[Amy.Sarin]

Robots.txt is a text (not html) file you put on your root directory to tell search robots which pages you would like them not to visit.
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it.

If you use robots.txt with your Sitemap, all major search engines (Google, Yahoo, MSN) will discover it automatically.

[carolinehill]

Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol. Robots.txt file usage is sometimes ignored. On the other hand, it is an important factor for the webpages being indexed properly and very easy to setup.

[bermuda]

If you are willing to give permission to the search engines crawling robots to access your web server in full, with all images, folders and contents being accessible, there is actually no need to make use of Robots.txt file and you can find thousands of websites which do not contain the item. But the real point happens when you would like to block some parts to get accessed.

For example, sometimes companies which have important information or several pictures folders on their servers, make use of Robots files to tell search engines not accessing the designated areas on their servers. Another example can be about shopping portals and naturally, a lot of their pages do not need to get indexed by search sites, so their admins block them through writing their URLs on the Robots files.

[Richard Smith]

Hi..

Robots.txt is a text file you put on your site to tell search engine, which page you would like to not visit by them. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site. It is not a kind of password protection and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door.
Here I would like to give an example like:you cannot prevent thieves from coming in but the good guys will not open to door and enter. Robots.txt is why we want to protect the some sensitive data from being indexed and displayed in search results.

Thanks
Search Engine Optimization NYC

[josephrex]

Robots.txt File is use for website crawling and indexing and You can tell Search Engines not to crawl your webpage through robots.txt

[Samuel]

we use this to hide some of the secret content on our site that we do not want to show visitors...

[pestindia86]

robot is called spider that spide your site and tell us which page you would like to spider or not

[Richard Smith]

Hello Friends..


Robots.txt is a text file that you need to put on your site to tell search robots which pages of your website you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door.


Thanks
Internet marketing Newyork (http://www.auroin.com)