Results 1 to 12 of 12
  1. #1

    What is Robots.txt?

    Hello Friends,

    Please tell me what is robots.txt.

  2. #2
    Senior Member
    Join Date
    Jul 2017
    Posts
    110
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  3. #3
    Registered User
    Join Date
    Feb 2017
    Posts
    670
    Robots Txt is an HTML attribute that is used to inform the search engines not to crawl and index the web pages in the website.

  4. #4
    Registered User
    Join Date
    Mar 2016
    Posts
    253
    Robots.txt is a file to give instructions to web robots about the website crawling; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  5. #5
    Registered User
    Join Date
    Dec 2015
    Location
    USA
    Posts
    418
    It is Robot txt and this means that there are certain places on the website where there is personal information and in these places customers are not allowed to go so in this case robot txt is used

  6. #6
    Registered User 24x7servermanag's Avatar
    Join Date
    Jul 2017
    Location
    India
    Posts
    213
    Robot.txt is used to crawl the pages website. It tells that which part of the area should not be accessed. We can define the pages which should not be accessed by putting the disallow tag in robot.txt. Those disallow pages are restricted to visit. It also help to index the web content.

  7. #7
    Member
    Join Date
    Jul 2017
    Location
    India
    Posts
    63
    robots.txt file is used to give instruction to the bots that crawl the website.

  8. #8
    Senior Member
    Join Date
    May 2017
    Posts
    227
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  9. #9
    Junior Member
    Join Date
    Jul 2017
    Posts
    25
    robot.txt is a file which let you communicate between the web crawler and website...in the way of telling which web pages to be crawled and which not to be

  10. #10
    Registered User
    Join Date
    Apr 2015
    Location
    New Delhi
    Posts
    676
    Thank For Sharing Valuable Information...

  11. #11
    Registered User
    Join Date
    Jan 2013
    Location
    India
    Posts
    55
    Robots.txt is a test file uploaded at the root of any website. It is used to allow or block URLs of the website to be crawled by different search engine. We need to follow a set protocol prescribed by robots.txt community

  12. #12
    Registered User
    Join Date
    Aug 2010
    Posts
    67
    A txt file that gives instructions to web crawlers what to do.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.