If Google does not fetch Robot txt file, is it web hosting problem or some other type of problem ?
If Google does not fetch Robot txt file, is it web hosting problem or some other type of problem ?
You can see the issue in the Webmaster Tool. BTW problem can be your side.
Maybe you place the file in the wrong place or named it wrong or some kind of other error can also exist but you can see the error in the GWT.
In some cases, a misconfigured firewall or DoS protection system (sometimes the site's content management system) block Googlebot from crawling the website. Protection systems are an important part of good hosting and are often configured to automatically block unusually high levels of server requests. However, because Googlebot often makes more requests than a human user, it can trigger these protection systems, causing them to block Googlebot and prevent it from crawling your website.
Make sure that in Settings → Privacy in the back-end you've checked Allow search engines to index this site and not the other one which is Ask search engines not to index this site.
Also, that was probably just a typo, but your robots.txt should look like this:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
█ Cheap VPS | $1 VPS Hosting
█ Windows VPS Hosting | Windows with Remote Desktop
█ Cheap Dedicated Server | Free IPMI Setup
Search engine automatically read robots.txt file, you have only submit your website sitemap.xml to Google webmaster tools.
Gbjazz.com | Novarustech.com | Id-it.ca | Seerairmiami.com | Baddiegalorefashion.com | Frontlinefp.com | Kayserlawgroup.com | Whitesailre.com | Keepingfamiliesconnected.com | Hdesigncenter.com | Proxifs.com | Alarmpa.com | Dq-construction.com | Midwifeutah.com | Danielwhiterealtor.com | Capitalbankcardoption.com
I think it comes under wrong category.
But robots plays important role in indexing and deindexing your website.
Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.
HostechSupport
24x7 Remote Services
Linux/Windows Server Administration Server Management
Get in touch: support@hostechsuppport.com
|
Bookmarks