Results 1 to 15 of 15
  1. #1

  2. #2
    Registered User shilpa's Avatar
    Join Date
    Jan 2017
    Posts
    81
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  3. #3
    Registered User
    Join Date
    Feb 2017
    Posts
    754
    Robot.txt is the process of informing search engines not to crawl and index the URL of the web page or website.

  4. #4

  5. #5
    Senior Member
    Join Date
    Dec 2016
    Posts
    155
    robots.txt is a standard used by websites to communicate with web crawlers and other web robots.

  6. #6
    Senior Member
    Join Date
    May 2015
    Posts
    647
    By robot.txt file you can prevent search engines from not crawling and indexing a webpage of your site that you do not like it to be indexed by a search engine. It is actually a search engine directive file.

  7. #7
    Registered User
    Join Date
    Aug 2016
    Location
    USA
    Posts
    44
    Robots.txt is a standard used by websites to communicate with web crawlers and other web robots. Robots.txt file is used to control what content search engines are allowed to access on your site. This is great for controlling duplicate content and for directing your crawl budget to your most important pages.

  8. #8
    Registered User
    Join Date
    May 2017
    Location
    Bangalore
    Posts
    63
    Use of Robots.txt:
    Robots.txt allowing access to precise crawlers
    Robots.txt Blocking specific folders/content
    Default Robots.txt

  9. #9
    Registered User Nekurokaze's Avatar
    Join Date
    Oct 2016
    Location
    Vietnam
    Posts
    6
    Based on my knowledge, robot.txt is like a blocker which will tell a particular search engine bot (in this case is Google bot) not to crawl into a particular page we don't want it to get in. It can be a good way to keep the bot from indexing some pages or links we don't want to index. However, since it only blocks the bot while the actual links still appear like normal, we can't use robot.txt as a way to "conceal" a site from other people.

  10. #10
    Senior Member
    Join Date
    Jun 2017
    Posts
    144
    Robots.txt file is a text file for restricting bots (robots, search engine crawlers ) from a website or certain pages on the website. Using a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or from certain folders and files.

  11. #11
    Junior Member
    Join Date
    Feb 2017
    Location
    CA
    Posts
    29
    The robots.txt file is used to provide instructions about the Web site to Web robots and spiders of search engine. Website owner can use robots.txt to keep cooperating Web robots from accessing all or parts of a Website that you want to not to crawl by search engines.

  12. #12
    Registered User
    Join Date
    Sep 2014
    Location
    Pennsylvania, USA
    Posts
    87
    Robot text is used to keep our visitor stay on the webpage

  13. #13
    Registered User
    Join Date
    Aug 2016
    Location
    22440 Ventura blvd Woodland Hills, California
    Posts
    53
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

  14. #14
    Registered User
    Join Date
    Sep 2016
    Posts
    153
    robots.txt is very important for any website to monitor

  15. #15
    Senior Member
    Join Date
    Mar 2017
    Location
    Bangalore
    Posts
    250
    In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.