What is robots.txt?

[seoinheritx]

Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.

[webcreations]

Robots.txt is a permissions file that can be used to control which webpages of a
website a search engine indexes. The file must be located in the root directory of
the website for a search engine website-indexing program (spider) to
reference.

[Cole001]

Robots.txt could be a document you set on your website to inform search robots that pages you'd like them to not visit. Robots.txt is by no means that necessary for search engines however typically search engines conform what they're asked to not do....

[borgninestallon]

Robot.txt is nothing but a text file where we put some rule to search engines not to crawl few pages with in a sites. If your site is 100% well structured then there is not need to use Robot.txt in your site.

[Engagedots]

robots.txt as a solution to help publishers control what content on their websites that Google's indexing spiders

[synapsesite]

Robots.txt is the text file in which we write few rule to give or restrict permissions to the crawlers.

[otonat007]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

[denniemark]

In short answer- A txt file that controls web robots.

[jamsen]

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

[titly555]

Hi
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.

[stephan07]

Robot.txt command use for crawling purpose of website. This technique uses for on-page optimization.

[Tikup]

It;s very important for all websites owners to have a good written Robots.txt file , because that file working with search engines .

[seoworldclass]

Hello,

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

By default search engine bots crawl everything possible unless they are forbidden from doing so. They always scan the robots.txt file before crawling the web site.

Declaring a robots.txt means that visitors (bots) are not allowed to index sensitive data, but it doesn't mean that they can't. The legal/good bots follow what is instructed to them, but the Malware robots don't care about it, so don't try to use it as a security for your web site

Thanks and Regards
seoworldclass.com

[terrijhon]

Well information is great..thanks

[webcreations]

The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.