Robots.txt

[advaitaahomes]

What is the use of robots.txt in SEO

[sajid israq]

There are many good reasons to stop the search engines from indexing certain directories on a website and allowing others for SEO purposes. Let's look at some examples.

Here's what you should do with robots.txt:

Take a look at all of the directories in your website. Most likely, there are directories that you'd want to disallow the search engines from indexing, including directories like /cgi-bin/, /wp-admin/, /cart/, /scripts/, and others that might include sensitive data.
Stop the search engines from indexing certain directories of your site that might include duplicate content. For example, some websites have "print versions" of web pages and articles that allow visitors to print them easily. You should only allow the search engines to index one version of your content.
Make sure that nothing stops the search engines from indexing the main content of your website.
Look for certain files on your site that you might want to disallow the search engines from indexing, such as certain scripts, or files that might contain email addresses, phone numbers, or other sensitive data.

[jane1]

Through adding robot.txt file you could give direction to search engines not to crawl your webpage in which you add it.

[Rajdeep Bose]

Robots.txt is the text file that is mostly used to instruct search engine which page should be crawled and which shouldn't be crawled.

[ramskl]

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[MitraVarghese]

Why we use Robot.txt file because from an SEO point of few we want to know that our file is crawling in Google or not and our which file is crawling in Google. You can check this by going to your site and checked it by adding/robot.txt to the URL. Let suppose your website is XYZ.com then you can add robot.txt by www.xyz.com/robot.txt. This will bring up a screen that will display the restrictions that you have placed on the ‘robots’ crawling your site.