What Is A Robots.txt File?

[ThedorisJackson]

Hi Friend,
Please can tell me about something about robots.txt file?

[rehal]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[AvniShergill]

Robots.txt helps to tell the web crawlers, which page of your website need to be crawled.

[joannapaul]

robots.txt is a file for search engines to know what are the pages to be crawled and indexed in your website.

[ehostingpk]

Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit.

[johnzakaria]

this text which manage the enter of Google spiders to your site, so be careful about this file and don't edit it if you understand how it works

[sanjay1]

robot.txt file is file which contain some instruction for search engines bot that how website or pages should be crawl or not, its mean these instruction tell bot how crawl or index your site.

[Sophiacalvin]

Hello

Web site owners use the /robots.txt file to give guidelines about their site to web robots; this is called The Robots Exemption Protocol. The "User-agent: *" means this segment applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

[stevex]

robots.txt tells to web robots what to read and which portion should not read.We can set privacy by allowing and disallowing which part of site we don't want to crawl.

[EdwardLee]

Robot.txt file is used to show search engine more about any website. Search engine gets an idea about which webpage has to crawl and which not. It is used to hide any web page from a search engine if you don't want to show it to others, it also used to prevent penalization.

[Glenn Rodgers]

Robots.txt file is telling or informing search engine spider not to crawl or visit this page. Just put robots.txt file in those pages where you wanna not allow to search engine crawl to visit or fetching your page's data.

[jannankhan1101]

Hello a robot.txt file tells a search engine bot which folder and pages should be crawled and indexed and which folders and pages should not be crawled/indexed. So its a good practice to use it as some times you will not want for search engine to index a specific webpage or web folder.

[Melissa Feeney]

Robots.txt file helps Google and other search engines in crawling and indexing your site. You can tell robots, which page of your website need to be crawled and which need to be avoided.

[crushmymugshot1]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

[interservermike]

robots.txt file is a text file that stops web crawler software, such as Googlebot, from crawling certain pages of your site. The file is essentially a list of commands, such Allow and Disallow, that tell web crawlers which URLs they can or cannot retrieve. So, if a URL is disallowed in your robots.txt, that URL and its contents won't appear in Google Search results.

Read more here.....
https://support.google.com/webmasters/answer/6062608?hl=en

[Adam Stark]

A robots.txt file is a text file that stops web crawler software, such as Googlebot, from crawling certain pages of your site. It is basically an Allow and Disallow button in your site for the crawler.

[rvtrainingchd]

Web site owners use the robots.txt file to give instructions about their site to Web spiders on how to crawl and index the website.

[christopher12]

Robots.txt: It's a simple text file that inform search engine bots how to crawl and index our website.

[Wooservers]

Robots.txt file is a text file that stops web crawler software, such as Googlebot, from crawling certain pages of your site.

[milos87popovic]

Robots.txt file is very important for your site, this is a convention to advising cooperating web crawlers and other web robots about accessing all or part of a website which is otherwise publicly viewable.

[Jesse R. Mitche]

In my view it is a mostly a file which helps crawler to know about your site. It tells crawler that which pages of your site need to be indexed.

[jayanta1]

A robots.txt file restricts access to your site by search engine robots that crawl the web. Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

[pryidevsblog]

robot.txt is file through which you can set the restriction which part of site you want to allow for the search engine crawlers and which one is not. The permitted part is neither seen in the search result nor indexed in search engine.

[anirban09P]

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.

[abhijepansd]

robots.txt file is useful for the search engines to crawls or not crawls the site pages. if your don't want to crawls the pages, you can use robots.txt file

day picnic near delhi (http://www.arounddelhi.net/?page_id=203) | picnic spots in gurgaon (http://www.kingfisheradventureretreat.com/picnic-spots.html)

[ilead]

A lot of definitions and information on robot.txt have already been given. I just want to add one of my personal observations. You must use robot.txt in your file even you do not want to block any page. The reason is simple when Google crawler comes to crawl your web pages, first of all it check robot.txt and sometimes (not always) it postpone the crawling for some time if it could not find any rebot.txt file attached.