Results 1 to 14 of 14
  1. #1

  2. #2
    Senior Member dennis123's Avatar
    Join Date
    Apr 2013
    Location
    Bangalore
    Posts
    2,273
    Hi Friends,
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  3. #3
    Registered User
    Join Date
    Jan 2019
    Posts
    292
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.

  4. #4
    Senior Member
    Join Date
    Oct 2018
    Posts
    306
    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

  5. #5
    Senior Member
    Join Date
    Nov 2018
    Posts
    653
    Robots.txt is a file that give instructions to the crawler as to which page to crawl and which page not to crawl. Any business if wants to retain the confidential information that an user has given from the crawler, then he can use Robots.txt file.

  6. #6
    Registered User
    Join Date
    Dec 2018
    Location
    51/1, Lalbagh Cross Road, Dhaka-1211
    Posts
    74
    Robots.txt is a text file that suggests robots (search engine bots)how to crawl & index pages on a website.

  7. #7

  8. #8

  9. #9
    Senior Member
    Join Date
    Jul 2019
    Posts
    103
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

  10. #10
    Senior Member
    Join Date
    Dec 2018
    Posts
    443
    if you want to use robots.txt for removing pages from indexing it will longer work hence google bots will now bypass robots.txt, use met no index

  11. #11
    Senior Member
    Join Date
    Jul 2006
    Location
    IndiaMDM
    Posts
    355
    Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.

  12. #12
    Senior Member
    Join Date
    Apr 2019
    Posts
    125
    robots.txt instruct robots to skip certain pages for getting indexed.

  13. #13
    robots.txt is a text file having certain instrunction which tell the google bot that which pages of the website should be crawl and index and which are not to crawl or index.

  14. #14
    Registered User
    Join Date
    Feb 2019
    Posts
    1,931
    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.