Results 1 to 15 of 15
  1. #1

  2. #2
    Registered User
    Join Date
    Jul 2017
    Location
    India
    Posts
    75
    it gives instruction to search engine spider that which pages of your sites you want to index and which pages want to ignore, there are many pages in website but not all useful for users so it's not important to index those pages in search engine, A robots.txt file tell to the robots which pages or parts of the website that should be index and which parts should not.

    an example of Robots.txt file is below:
    (!) User-agent: Googlebot/MSNbot/yahoobot.
    Disallow: add your link here which do not want to index.

    (!!) User-agent:*
    Disallow: /

    (!!!)(*) indicates all the search engines.
    (/) indicates all pages of a site.

  3. #3
    Registered User
    Join Date
    Aug 2017
    Location
    United States
    Posts
    1
    Hi I am Bmarl from United States and I work with one of the most reputed company named - Ancientwriters who provide the perfect writing as well as publishing services . I have been working there from past few years. If you are looking for such writing and publishing services then you can visit their website.

  4. #4
    Junior Member
    Join Date
    Jul 2017
    Posts
    23
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

  5. #5
    Member
    Join Date
    Jul 2017
    Posts
    87
    robots.txt is a text file its helps to block all web crawler from all web pages or specific web crawler from specific web pages.

  6. #6
    Registered User
    Join Date
    Jan 2014
    Location
    United Kingdom
    Posts
    192
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  7. #7
    Registered User
    Join Date
    Feb 2017
    Posts
    738
    Robots.txt is a file which is used by webmasters to instruct search engine robots not to crawl the web pages in the website.

  8. #8

  9. #9
    Registered User
    Join Date
    Dec 2012
    Location
    India
    Posts
    758
    Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

  10. #10
    Member
    Join Date
    Jul 2017
    Posts
    49
    Robots.txt is a special kind of file that is not HTML or any other type. The purpose of having a robots.txt file is to tell the search engine that what content on your site is allowing search bots and indexes.

  11. #11
    Registered User
    Join Date
    Sep 2016
    Posts
    213
    robots.txt is text file. it helps to block all web crawler from all pages or specific web crawler form specific web page.

  12. #12

  13. #13
    Member
    Join Date
    Feb 2016
    Location
    Beauty Exchange Center,No.121, Guangyuan West Road,Guangzhou,China,51000
    Posts
    98
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

  14. #14
    Registered User
    Join Date
    Dec 2015
    Location
    USA
    Posts
    601
    It is Robot txt and this means that there are certain places on the website where there is personal information and in these places customers are not allowed to go so in this case robot txt is used

  15. #15
    Registered User 24x7servermanag's Avatar
    Join Date
    Jul 2017
    Location
    India
    Posts
    515
    Server Management Company
    India's Leading Managed Service Provider | Skype: techs24x7
    Cpanel Technical Discussions - Lets talk !

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.