Hello friends,
Where robots.txt file uploaded to hosting directory?
Printable View
Hello friends,
Where robots.txt file uploaded to hosting directory?
Robots.txt is a text file located in the site's root directory that specifies for search engines' crawlers and spiders what website pages and files you want or don't want them to visit.
Add robots.txt file to the root directory of your website.Quote:
Originally Posted by vashilove
Format and location rules for the robots.txt file:
The file must be named robots.txt
Your site can have only one robots.txt file.
The robots.txt file must be located at the root of the website host that it applies to. For instance, to control crawling on all URLs below http://www.example.com/, the robots.txt file must be located at http://www.example.com/robots.txt. It cannot be placed in a subdirectory ( for example, at http://example.com/pages/robots.txt). If you're unsure about how to access your website root, or need permissions to do so, contact your web hosting service provider. If you can't access your website root, use an alternative blocking method such as meta tags.
A robots.txt file can apply to subdomains (for example, http://website.example.com/robots.txt) or on non-standard ports (for example, http://example.com:8181/robots.txt).
If you are hoping that all crawlers and bots are going to respect what you are going to put on robots.txt, then you will be disappointed. Only a few of those many bots even care where you put robots.txt or what you put into
But Google and other popular search engines are believed to respected a properly placed robots.txt ( in the root folder ). Good luck.