Page 2 of 2 FirstFirst 12
Results 16 to 18 of 18
  1. #16
    Member
    Join Date
    Sep 2018
    Posts
    74
    The robots.Txt file is a part of the REP, a group of web necessities that control how robots crawl the web, access and index content material, and serve that content material up to users. These crawlers advised to disallow or allow the user agents.
    It is not necessary in your website.

  2. #17
    Yes it is essential, as it is a component of on page optimization, so it is a must do thing to add robots file in your website, with the help of robots file you can control deindexing of any pages and can tell google and other search engine bots that which directories are private and should not be indexed.

  3. #18
    Senior Member
    Join Date
    Feb 2016
    Location
    Bangladesh
    Posts
    486
    Robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.

Page 2 of 2 FirstFirst 12

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.