Results 1 to 13 of 13
  1. #1

  2. #2
    Registered User
    Join Date
    Feb 2019
    Posts
    1,587
    The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  3. #3
    Senior Member
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    4,326
    Disallow implies that the webpages listed are restricted from search engine crawling.
    Cheap VPS | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | Free IPMI Setup

  4. #4
    Junior Member
    Join Date
    Jul 2019
    Posts
    10
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  5. #5
    Senior Member
    Join Date
    Jul 2018
    Posts
    380
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  6. #6
    Registered User
    Join Date
    Apr 2016
    Posts
    655
    Web site owners use the /robots.txt file to give instructions about their site to search engine robots. The "Disallow: /" tells the search engine bots that it should not visit any pages on the site.

  7. #7
    Senior Member
    Join Date
    Nov 2018
    Posts
    564
    A robots.txt file allows you to restrict the access of search engine crawlers to prevent them from accessing specific pages or directories. They also point the web crawler to your page’s XML sitemap file.

  8. #8
    Member
    Join Date
    May 2019
    Posts
    95
    It tells the robot that it should not visit any pages on the website.

  9. #9
    Registered User
    Join Date
    Feb 2019
    Posts
    1,587
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

  10. #10
    Registered User techwhizz's Avatar
    Join Date
    Jul 2019
    Location
    Florida
    Posts
    2
    The "Disallow: /" tells the robot that it should not visit any pages on the site. It is "User-agent: *" means this section applies to all robots.

  11. #11
    Member
    Join Date
    May 2019
    Posts
    60
    The "Disallow: /" tells the robot that it should not visit any pages on the site.

  12. #12

  13. #13
    Senior Member
    Join Date
    Jan 2019
    Location
    Ghaziabad
    Posts
    207
    Disallow in the robots.txt file contains the information for the Search Engine Crawlers what URL, Directory or Folder is restricted for crawling.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.