What is the use of robots.txt?

[artijain1900]

Hello Guys,


I would like to know that What is the use of robots.txt? tell me ..

[othername0104]

For full information on the robots.txt standard, go to: http://www.robotstxt.org/wc/robots.html

[astuartspindlow]

Robots.txt is a file which is used to tell the search engine which page has to be crawled or not by allowing or disallowing.

[amirali]

robots.txt file is a simple text file to give instructions about their site to web robots, this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all search engine robots. The "Disallow: /" tells the robot that it should not visit any pages on the website.

[ShreyaKoushik]

Use of Robots.txt -

The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

[Liyanscom321]

Nice Info.
Thanks for sharing.

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[benben1]

-You must save your robots.txt code as a text file,
-You must place the file in the highest-level directory of your site (or the root of your domain), and.

[longcanlc]

-You must save your robots.txt code as a text file,
-You must place the file in the highest-level directory of your site (or the root of your domain), and.

Can you help me how to create a Robot.txt?

[Williams Reus]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

[alex.thomson]

You might be surprised to hear that one small text file, known as robots.txt, could be the downfall of your website. If you get the file wrong you could end up telling search engine robots not to crawl your site, meaning your web pages won’t appear in the search results. Therefore, it’s important that you understand the purpose of a robots.txt file and learn how to check you’re using it correctly.