What is robots.txt?

[cookaltony]

Robots.txt is one of the best text file for deny the web pages to crawl in the search engine.it is beneficial to protect your important pages in the search engine for crawling.

[theavi]

Robots.txt file is a notepad file which instruct to search engine spiders to not to crawl the pages which this file has.

[thearav]

Robots.txt is a file which instruct google to not to crawl the pages which this file has. The robots.txt file should be on server in root directory of website.

[bivin007]

Robots.txt is a text file located in a website’s root directory that specifies what website pages and files you want (or don’t want) search engine crawlers and spiders to visit. Usually, website owners want to be noticed by search engines; however, there are cases when it’s not needed.

[taxiongo]

A robots.txt file is a set of instructions for bots. This file is included in the source files of most websites. Robots.txt files are mostly intended for managing the activities of good bots like web crawlers, since bad bots aren't likely to follow the instructions.

[tbsind]

We can say that Robots.txt is a file where you can create allow to control the crawling of your website. It allows search engines to choose which website url they can crawl or not.