PDA

View Full Version : What is robots.txt?



seoinheritx
11-26-2012, 03:02 AM
Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.

webcreations
11-26-2012, 03:16 AM
Robots.txt is a permissions file that can be used to control which webpages of a
website a search engine indexes. The file must be located in the root directory of
the website for a search engine website-indexing program (spider) to
reference.

Cole001
11-26-2012, 04:54 AM
Robots.txt could be a document you set on your website to inform search robots that pages you'd like them to not visit. Robots.txt is by no means that necessary for search engines however typically search engines conform what they're asked to not do....

borgninestallon
11-26-2012, 05:01 AM
Robot.txt is nothing but a text file where we put some rule to search engines not to crawl few pages with in a sites. If your site is 100% well structured then there is not need to use Robot.txt in your site.

Engagedots
11-26-2012, 05:50 AM
robots.txt as a solution to help publishers control what content on their websites that Google's indexing spiders

synapsesite
11-26-2012, 06:32 AM
Robots.txt is the text file in which we write few rule to give or restrict permissions to the crawlers.

otonat007
11-26-2012, 08:14 AM
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

denniemark
11-26-2012, 08:19 AM
In short answer- A txt file that controls web robots.

jamsen
11-26-2012, 07:58 PM
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

titly555
11-29-2012, 07:20 AM
Hi
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website.

stephan07
11-29-2012, 07:33 AM
Robot.txt command use for crawling purpose of website. This technique uses for on-page optimization.

Tikup
11-29-2012, 09:20 AM
It;s very important for all websites owners to have a good written Robots.txt file , because that file working with search engines .

seoworldclass
11-29-2012, 02:10 PM
Hello,

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

By default search engine bots crawl everything possible unless they are forbidden from doing so. They always scan the robots.txt file before crawling the web site.

Declaring a robots.txt means that visitors (bots) are not allowed to index sensitive data, but it doesn't mean that they can't. The legal/good bots follow what is instructed to them, but the Malware robots don't care about it, so don't try to use it as a security for your web site

Thanks and Regards
seoworldclass.com

terrijhon
11-30-2012, 01:17 AM
Well information is great..thanks

webcreations
11-30-2012, 01:30 AM
The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

arindamdutta16
12-03-2012, 05:16 AM
The robots.txt is a simple text file in your web site. Search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

blessy_smith
12-04-2012, 02:00 AM
The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

abhirampathak3
12-04-2012, 02:07 AM
Simply I acn say, The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

mikeslough5
12-04-2012, 02:33 AM
Robots text file forced to search engine how to crawl and index your site or web pages. Some times we don't want search engine read our under develop pages so we use Robot Text files.

Zora2012
12-04-2012, 05:20 AM
Robots.txt is a file we'll place on your website which asks that specific files or pages on your website be ignored by search engines. Excluding such files or pages from the search engines could be as a result of a preference for privacy, or that the content on those files or pages isn't relevant to your site categorization as a whole. Implementing a robots.txt file does not guarantee that these files or pages will not appear in search engine result pages, it's more of a way of telling the search engines that "you can ignore these these files and pages over here."

muzz
12-04-2012, 07:02 AM
robot.txt gives you an option which data you want to index and which is not.