what is importance of robot.txt in seo?

[noahwilson]

Hello Guys,
Please tell me what is importance of robot.txt in seo? i am new in seo field i want to know more about it, so please share your reviews about its and how can be make robot.txt useful for seo.
Thanks.

[mobileweb003]

Robots.txt file is in the root directory of a website which has been created to direct the activity of search engine crawlers or spiders.

[Vamp1re]

Robots text file are those which forced to search engine how to crawl and index your site or web pages.
It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

[Markyarnell08]

It helps in directing search engine spiders to follow and index contents found on your site.

[SASA techno]

Robot.txt is the notepad file you have a crawl you page which one that mostly not use so you add that one page in robot.txt so that page is online so client see very well but Google is not crawl that one page....

[ackerlee]

Robot.txt is a file which contains the instruction for the crawler or bots of search engines. It instruct the search engine crawler about the web pages which should they crawl and which do not.

[JerrickYeoh]

You can set your robot.txt in your webmaster tools, it usually largely use by webmaster to set that which URL to follow and index and which URL not to follow and index to maintain their site quality .

[nrvaghela]

Robot.txt is a file which prevents files, folders from being cached in search engine. You can specify files and folders that are not to be crawled in search engine for security purpose you must do it.

[sabrinasai]

It provide the search engine crawlers permission to follow or index particular page or content or image or link of your web page.

[aryanmarshall45]

Robot.txt file give instruction to google crawler or robot which file not to index or crawl. Basically robot.txt file restrict google crawler to crawl written page in file.

[seoworldclass]

Hello Friends,

Robot.txt files tell your instructions to a search engine robot. It depends on what you what it to do. If you don't have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. These files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. It is also important to make a note that it is not a method that prevents search engines from crawling your site. So it is important to create a robot.txt file

Thanks and Regards
seoworldclass.com

[James-A]

There are many search engine are available on the web. When a search engine crawler comes to the site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

[jamsen]

It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sensitive data, it is too naive to rely on robots.txt to protect it from being indexed and displayed in search results.

[johnirron]

As per my info; Robot.txt is extremely helpful to stop the google crawlers where you do not need them to cache your WebPages or admin pages.

[brainpulse]

robot.txt is basically a file which use to stop Google's crawler to visit any particular pages which site administrator not want to read by Crawler.

[limanto]

Robots.txt is such an important document to be added to a website. This site has it too! Mainly I use it to tell the SE that I eaither want (or not) to index a website and how (based on folders I want to allow or restrict).

[blessy_smith]

It provide the search engine crawlers permission to follow or index particular page or content or image or link of your web page.

[Manov]

The robots.txt file is what you would use to tell the search engines what files to disallow - NOT to index...

Here is a good article explaining what a robots.txt file is and on how to set up your own robots.txt file for your websites.

http://www.free-seo-news.com/all-about-robots-txt.htm

hope it useful..:)

[GeorgeS2]

Robot.txt is important file for website, this file command the indexing bot for indexing or stop indexing website or page or any particular section of website.

[SANTOSH123]

If we want to hide any information that information google not read we can use robot.txt.

[mitsay]

robot.txt file is for bot, tell him what is allowed and disallow for indexation

[innovativeweb]

Hello,

Thank you very much.Tips are really helpful.Thank you very for sharing your views with us.

[webcreations]

Robots.txt files inform search engine spiders how to interact with indexing your content. By default search engines are greedy.They want to index as much high quality information as they can & will assume they can crawl everything unless you tell them otherwise. Robots.txt is often found in your site’s root directory and exists to regulate the bots that crawl your site. This is where you can grant or deny permission to all or some specific search engine robots to access certain pages or your site as a whole.This file is used to communicate with search engine spiders telling them what they cannot have access to.

[limanto]

I was wondering what that was. Thanks guys.

[akashram]

Robot.txt is great when search engines frequently visit your site and index your content.

[Gurmeet]

The main importance of Robots.txt files is that it tells the search engines which pages to access and index on your website on which pages not to. For example, if you specify in your Robots.txt file that you don’t want the search engines to be able to access your thank you page, that page won’t be able to show up in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.

[brandappz]

Robots.txt is a simple text (not html) file you put on your website root directory to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all.

[Priya321]

thnx to all

[SANTOSH123]

Robot.txt file give instruction to google crawler which file not to index or crawl.