Printable View
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth.
Robot.txt means which one file that not crawling in search engine that file is online visitor see that but search engine that file is not crawl.
It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. A robots.txt file restricts access to your site by search engine robots that crawl the web. Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.