Where do you put your robots.txt?
Where do you put your robots.txt?
Robots.txt. While simple, a robots.txt file can have a large effect on how search engines crawl your website.
Keep in mind that only one file or folder can be used per Disallow line. You may add as many Disallow lines as you need. Once complete, save and upload your robots.txt file to the root directory of your site. For example, if your domain is www.mydomain.com, you will place the file at www.mydomain.com/robots.txt.
I think the better way to describe it is to have it in the root web folder of your domain.
so http://example.com/robots.txt you can also put your sitemap.xml in the root or refer to it with a Sitemap: http://example.com/fldr/smap.xml line in your robots.txt.
You can use Google Webmaster Tools to check to make sure you haven't restricted anything you didn't mean to.
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.
Domain Domains
In the domain root of the website, we put robots.txt & sitemap.xml both files.
Gbjazz.com | Novarustech.com | Id-it.ca | Seerairmiami.com | Baddiegalorefashion.com | Frontlinefp.com | Kayserlawgroup.com | Whitesailre.com | Keepingfamiliesconnected.com | Hdesigncenter.com | Proxifs.com | Alarmpa.com | Dq-construction.com | Midwifeutah.com | Danielwhiterealtor.com | Capitalbankcardoption.com
Robots.txt file is a text file which provides instruction to how to crawl and index page or content of the website.
|
Bookmarks