PDA

View Full Version : Where robots.txt file uploaded to hosting directory?



vashilove
05-16-2018, 02:55 AM
Hello friends,

Where robots.txt file uploaded to hosting directory?

littlemgrnoida
05-16-2018, 09:00 AM
Robots.txt is a text file located in the site's root directory that specifies for search engines' crawlers and spiders what website pages and files you want or don't want them to visit.

evejones
05-16-2018, 11:53 PM
Hello friends,

Where robots.txt file uploaded to hosting directory?

Add robots.txt file to the root directory of your website.

hwrank2018
05-17-2018, 01:49 AM
Format and location rules for the robots.txt file:

The file must be named robots.txt
Your site can have only one robots.txt file.
The robots.txt file must be located at the root of the website host that it applies to. For instance, to control crawling on all URLs below http://www.example.com/, the robots.txt file must be located at http://www.example.com/robots.txt. It cannot be placed in a subdirectory ( for example, at http://example.com/pages/robots.txt). If you're unsure about how to access your website root, or need permissions to do so, contact your web hosting service provider. If you can't access your website root, use an alternative blocking method such as meta tags.
A robots.txt file can apply to subdomains (for example, http://website.example.com/robots.txt) or on non-standard ports (for example, http://example.com:8181/robots.txt).

dilipcybex
05-17-2018, 03:14 AM
If you are hoping that all crawlers and bots are going to respect what you are going to put on robots.txt, then you will be disappointed. Only a few of those many bots even care where you put robots.txt or what you put into

But Google and other popular search engines are believed to respected a properly placed robots.txt ( in the root folder ). Good luck.