Why Is Robots txt Important..?

[swatijain2233]

Hello friends,

I would Like to know that Why Is Robots txt Important..?

[milinmaru]

Robots.txt is the common name of a text file that is uploaded to a website’s root directory and linked in the HTML code of the website.

The robots.txt file is used to provide instructions about the website to web robots and spiders. Web authors can use robots.txt to keep cooperating web robots from accessing all or parts of a web site that you want to keep private.

[Snehap23]

Robots.txt is one file for the crawler to prevent the website to crawl.

[vinukum]

Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Basic format of the robot.txt file.
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

[Jessicad0505]

Robots.txt is used primarily to manage crawler traffic to your site, and occasionally to keep a page off Google, depending on the file type.

[Saravanan28]

The robots.txt file is used to provide instructions about the website to web robots and spiders. Web authors can use robots.txt to keep cooperating web robots from accessing all or parts of a web site that you want to keep private.

[dombowkett]

Robots.txt file send instructions to Google that which part should be indexed or not.