Please give me the answer.
I think this file name that controls the crawler to visit your that page of website which is under programing
Robots are search engine crawler. They are useful for fetching the data from the website.
Robots is a catch-all, or generic term for programs and automated scripts that "crawl" through the web (the Internet) and collect data from websites and anything else on the Internet that they can find.
Thanks a lot
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sensitive data, it is too naive to rely on robots.txt to protect it from being indexed and displayed in search results.
Three-link way is a process that is used for online site optimization as it can widen the scope of links that can be directed to your own page.
|Find Web Hosting|
|Shared Web Hosting||UNIX & Linux Web Hosting||Windows Web Hosting||Adult Web Hosting|
|ASP ASP.NET Web Hosting||Reseller Web Hosting||VPS Web Hosting||Managed Web Hosting|
|Cloud Web Hosting||Dedicated Server||E-commerce Web Hosting||Cheap Web Hosting|
|Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.|