Page 1 of 2 12 LastLast
Results 1 to 15 of 19
  1. #1

  2. #2
    Senior Member
    Join Date
    Nov 2018
    Posts
    1,348
    A robots.txt shows which pages or files the Googlebot can or can't request from a website. Webmasters usually use this method to avoid overloading the website with requests.

  3. #3
    Senior Member
    Join Date
    Jul 2006
    Location
    IndiaMDM
    Posts
    365
    Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.
    HostechSupport
    24x7 Remote Services
    Linux/Windows Server Administration Server Management
    Get in touch: support@hostechsuppport.com

  4. #4
    Registered User
    Join Date
    Jun 2018
    Posts
    1,416
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.

  5. #5
    Registered User
    Join Date
    Jan 2020
    Location
    USA
    Posts
    409
    Robots.txt send instructions to google for website indexing.

  6. #6
    Senior Member
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    5,019
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | Free IPMI Setup

  7. #7
    Registered User
    Join Date
    Feb 2020
    Posts
    77
    is a text file that tells web robots which pages on your webpage to creep. It likewise advises web robots which pages not to slither. Suppose an internet searcher is going to visit a website.

  8. #8
    Registered User
    Join Date
    Jul 2019
    Posts
    40
    The robots.txt file is part of the robots exclusion protocol (REP), a group of web

  9. #9

  10. #10
    Robots.txt files control crawler access to certain areas of your site. While this can be very dangerous if you accidentally disallow Googlebot from crawling your entire site (!!), there are some situations in which a robots.txt file can be very handy.

    Some common use cases include:

    Preventing duplicate content from appearing in SERPs (note that meta robots is often a better choice for this)
    Keeping entire sections of a website private (for instance, your engineering team’s staging site)
    Keeping internal search results pages from showing up on a public SERP
    Specifying the location of sitemap(s)
    Preventing search engines from indexing certain files on your website (images, PDFs, etc.)
    Specifying a crawl delay in order to prevent your servers from being overloaded when crawlers load multiple pieces of content at once

  11. #11
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
    Last edited by discusshostingadmin; 04-07-2020 at 06:23 AM.

  12. #12
    Senior Member
    Join Date
    Dec 2019
    Posts
    1,837
    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  13. #13
    Registered User
    Join Date
    Feb 2020
    Posts
    77
    The robots. txt document is a book record that tells web robots which pages on your website to slither. It likewise advises web robots which page not to slither. Suppose a web crawler is going to visit a webpage.

  14. #14
    Senior Member
    Join Date
    Mar 2020
    Posts
    1,214
    Robots. txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots. txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

  15. #15
    Senior Member
    Join Date
    Aug 2015
    Posts
    116
    its helpfull to provide the info to google about your targeting url and block by your end.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.