Results 1 to 12 of 12
  1. #1

    What’s the significance of Robots.txt file in a website?

    Hlo Friends,
    Can anyone tell me, What’s the significance of Robots.txt file in a website?

  2. #2
    Senior Member
    Join Date
    Aug 2015
    Posts
    116
    Robots.txt file is most important file in seo for a website. Whenever crawler crawl a website it search for robots file to check any instructions.

  3. #3
    Senior Member
    Join Date
    Feb 2017
    Posts
    219
    Robots.text file tell the search engine which file is indexed or which is not.

  4. #4

  5. #5
    Senior Member
    Join Date
    Mar 2017
    Location
    Bangalore
    Posts
    250
    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

  6. #6
    Registered User
    Join Date
    Mar 2017
    Posts
    37
    The robots.txt is a normal text file placed on your web server which tells Google bot if they should access your file or not.

  7. #7
    Registered User
    Join Date
    Jul 2016
    Posts
    256
    Robots.txt file is important because you can easily crawl web page by giving permission to search engine.

  8. #8
    Senior Member
    Join Date
    Feb 2016
    Location
    Bangladesh
    Posts
    486
    Robots.txt file helps a website to index on the search engine and it continuously crawls the different data for that website.

  9. #9
    Registered User
    Join Date
    Feb 2017
    Posts
    754
    Well, robots.txt is a file which is used to inform the search engines about the crawling and indexing of the web page or website. You can block or allow the search engine bots to crawl your website.

  10. #10
    Senior Member sadianisar's Avatar
    Join Date
    May 2016
    Posts
    2,781
    robots.txt is a file in which we instruct web crawler to not visit and index the link or page or website. It is an essential On-Page SEO element used for newly built websites.

  11. #11
    Registered User
    Join Date
    Apr 2017
    Posts
    107
    Robot.txt is a file which is actually a text file created to instruct robots (search engine robots) that how and which pages to crawl of a website.

  12. #12
    Registered User
    Join Date
    Mar 2017
    Posts
    39
    It like as file permission access

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.