What is a robots.txt file?

[Sophiacalvin]

What is a robots.txt file?

[jayanta1]

The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Creating a robots.txt file can actually improve your website indexation.

[akashram]

A robots.txt file is a text file that tells web crawler to crawling certain pages of website. The file is essentially a list of commands, such Allow and Disallow , that tell web crawlers which URLs they can or cannot retrieve.

[AaliyahLiverpol]

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

[Melissa Feeney]

Robots.txt file is for Google bots, it helps to tell them, which webpage of your website need to be crawled and which should be avoided. It is always advisable to check the robots.txt of the website before any submission.

[bruceacacio]

A robots.txt file is a text file that tells web crawler to crawling certain pages of website. The file is essentially a list of commands, such Allow and Disallow , that tell web crawlers which URLs they can or cannot retrieve.

Yes I agree with you,Its a file having the list of commands who tell the search engine crawler to crawling certain pages of website.

[edwina]

It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.

[soulspotter]

Robot.txt file is a file through this file you can set the authorization for the search engine spider what they have to crawl and what not. if you are not allow the crawler to crawler your certain area of website than you can disallow robot via robot.txt.

[divorceclass]

Robot.txt file used for Allow and Disallow on site.

[nancyisabell]

A robots.txt document is a content record that stops web crawler programming, for example, Googlebot, from creeping certain pages of your website. The record is basically a rundown of summons, such Allow and Disallow , that tell web crawlers which Urls they can or can't recover.

[Pravin]

With Robot.txt file you can block the web pages that you do not want bots to consider for crawling or indexing. If it is not made in a proper way, you may block the whole site from crawling which could create a big problem. It is always preferred to get it done by someone who has got a proper idea on how robot.txt is made. Or you can simply use robot.txt file file option in webmaster tools to know whether is is created properly or has got any issue.

[jaysh4922]

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

[Jackandrew]

The robots.txt file is a simple txt file that are placed on your server, if you go www.domain.com/robots.txt you see the file of websites that the site owner is asking the search engines to "skip" (or "disallow"). If any files and directories (which hurts your business) you don't want indexed by search engines, you can use a robots.txt file.

[ATSI]

robots file has a .txt extension that contains instructions for crawlers whether to crawl the website and index its pages

[nathankian]

Robots.txt file to control which pages of your site are indexed by search engines.

[john2]

Robot.txt file may be used to follow or not follow the links during crawling.

[webcreations]

Robots.txt is the file that are used to define rules to instruct robots to not crawl certain files and directories within your site.

[Rajdeep Bose]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

[vcominfotech]

Robots.txt file is very very important to a website.The robots.txt is to inform the spider to dont crawl the particular page in the website.

[acsius1]

Robots.txt files inform search engine spiders how to interact with indexing your content.

[nilam-shah]

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.