Results 1 to 8 of 8
  1. #1
    Registered User
    Join Date
    Nov 2018
    Posts
    8

    Where do you put your robots.txt?

    Where do you put your robots.txt?

  2. #2
    Senior Member
    Join Date
    Jul 2018
    Posts
    280
    Robots.txt. While simple, a robots.txt file can have a large effect on how search engines crawl your website.

  3. #3
    Registered User
    Join Date
    Jul 2018
    Posts
    199
    Keep in mind that only one file or folder can be used per Disallow line. You may add as many Disallow lines as you need. Once complete, save and upload your robots.txt file to the root directory of your site. For example, if your domain is www.mydomain.com, you will place the file at www.mydomain.com/robots.txt.

  4. #4
    Registered User
    Join Date
    Nov 2018
    Location
    USA
    Posts
    22
    I think the better way to describe it is to have it in the root web folder of your domain.
    so http://example.com/robots.txt you can also put your sitemap.xml in the root or refer to it with a Sitemap: http://example.com/fldr/smap.xml line in your robots.txt.
    You can use Google Webmaster Tools to check to make sure you haven't restricted anything you didn't mean to.

  5. #5
    Registered User
    Join Date
    Jan 2018
    Posts
    546
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  6. #6
    Registered User
    Join Date
    Apr 2016
    Location
    Greece
    Posts
    555
    Quote Originally Posted by tmblearning View Post
    Where do you put your robots.txt?

    Open a notepad and save the file as robots, all lowercase. Make sure to choose .txt as the file type extension (in word, choose 'plain text').

    Next step is, add the following two lines of text to your file are as given below:
    User-agent:*
    Disallow:
    Domain Domains

  7. #7

  8. #8
    Member
    Join Date
    Dec 2017
    Posts
    57
    Robots.txt file is a text file which provides instruction to how to crawl and index page or content of the website.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.