What is robots.txt and why we use robots.txt?

[angelinagertz]

What is robots.txt and why we use robots.txt?

[fayeseom]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[eCourierz]

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers.

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[ShreyaKoushik]

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

[amanshastry43]

Robot.txt are HTML files that inform the crawler that this page of site is not to crawl or index

[david hong]

Robots.txt file is used to guide the search engine automatically to the page you want it to search for and then index the page. Most sites also have the directories and files do not need to search engine robots to visit

[rajendrakholgad]

Robots.txt file is helps google or other search engine to crawl website. This file allows search engine to crawl or block specific pages of website.

[Zopsoft]

Its a set of instruction for robots about what to crawl and what to not. You can also restrict the type of bots visiting your website.

There's an another thread discussing the same thing. For those who are looking for more information can visit :

http://forums.hostsearch.com/showthread.php?94913-Why-we-use-Robots-txt-File

[Jeffwilliams]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

You can also hire a good SEO firm like SeoTuners to optimized your website.

[salenaadam]

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

[sadianisar]

robots.txt is important in SEO if the website is newly built. It is a file in which we write instructions for crawlers to not visit and index the link or webpage.