what is importance of robot.txt in seo?

[limanto]

Robots.txt is such an important document to be added to a website. This site has it too! Mainly I use it to tell the SE that I eaither want (or not) to index a website and how (based on folders I want to allow or restrict).

[blessy_smith]

It provide the search engine crawlers permission to follow or index particular page or content or image or link of your web page.

[Manov]

The robots.txt file is what you would use to tell the search engines what files to disallow - NOT to index...

Here is a good article explaining what a robots.txt file is and on how to set up your own robots.txt file for your websites.

http://www.free-seo-news.com/all-about-robots-txt.htm

hope it useful..

[GeorgeS2]

Robot.txt is important file for website, this file command the indexing bot for indexing or stop indexing website or page or any particular section of website.

[SANTOSH123]

If we want to hide any information that information google not read we can use robot.txt.

[mitsay]

robot.txt file is for bot, tell him what is allowed and disallow for indexation

[innovativeweb]

Hello,

Thank you very much.Tips are really helpful.Thank you very for sharing your views with us.

[webcreations]

Robots.txt files inform search engine spiders how to interact with indexing your content. By default search engines are greedy.They want to index as much high quality information as they can & will assume they can crawl everything unless you tell them otherwise. Robots.txt is often found in your site’s root directory and exists to regulate the bots that crawl your site. This is where you can grant or deny permission to all or some specific search engine robots to access certain pages or your site as a whole.This file is used to communicate with search engine spiders telling them what they cannot have access to.

[limanto]

I was wondering what that was. Thanks guys.

[akashram]

Robot.txt is great when search engines frequently visit your site and index your content.

[Gurmeet]

The main importance of Robots.txt files is that it tells the search engines which pages to access and index on your website on which pages not to. For example, if you specify in your Robots.txt file that you don’t want the search engines to be able to access your thank you page, that page won’t be able to show up in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.

[brandappz]

Robots.txt is a simple text (not html) file you put on your website root directory to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all.

[Priya321]

thnx to all

[SANTOSH123]

Robot.txt file give instruction to google crawler which file not to index or crawl.