Results 1 to 6 of 6

Thread: Robot txt file

  1. #1
    Senior Member
    Join Date
    May 2014
    Posts
    217

    Robot txt file

    If Google does not fetch Robot txt file, is it web hosting problem or some other type of problem ?

  2. #2
    Senior Member
    Join Date
    Mar 2018
    Location
    Delhi
    Posts
    184
    You can see the issue in the Webmaster Tool. BTW problem can be your side.

  3. #3
    Senior Member
    Join Date
    Mar 2017
    Posts
    373
    Maybe you place the file in the wrong place or named it wrong or some kind of other error can also exist but you can see the error in the GWT.

  4. #4
    Registered User
    Join Date
    Feb 2014
    Location
    delhi
    Posts
    515
    In some cases, a misconfigured firewall or DoS protection system (sometimes the site's content management system) block Googlebot from crawling the website. Protection systems are an important part of good hosting and are often configured to automatically block unusually high levels of server requests. However, because Googlebot often makes more requests than a human user, it can trigger these protection systems, causing them to block Googlebot and prevent it from crawling your website.

  5. #5
    Registered User
    Join Date
    Jan 2018
    Posts
    188
    Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.

    Also, that was probably just a typo, but your robots.txt should look like this:

    User-agent: *

    Disallow: /wp-admin/
    Disallow: /wp-includes/

  6. #6
    Senior Member
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    3,244
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Hosting | Free IPMI Setup

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.