what is ue of robots.txt

[rajeshedunomcon]

what is use of robots.txt

[Ruby]

there are some portions on your web page or some times the pages that you don't want scrawlers to crawl, so we use robots file. Also, there are certain characters excluding hifen and pipes that stops or inhibits crawling. So, you can use robots file here and it will boost the crawling process. It is not advisable to use robots to stop crawling, you can use it but don't use very frequently. Use when necessary.

[Rado_Ch]

The robots.txt is basically the instructions you give to web crawlers about your website. Indeed you can use it to tell the crawlers not to visit certain pages of your website, but this has to be done very carefully. Disallowing certain pages might lead to your whole website being hidden from robots. Also keep in mind that this file is publicly available, so you cannot use it to hide information from your website globally. Another thing to have in mind is that not all robots will follow the instructions of the robots.txt file - generally there are many malware and spam oriented bots that will still crawl pages that you disallowed.

All in all be very certain when you use robots.txt because it might lead to much bigger issues if not set properly

[milos87popovic]

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to advising cooperating web crawlers and other web robots about accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.