Page 1 of 2 12 LastLast
Results 1 to 15 of 21
  1. #1
    Registered User
    Join Date
    Dec 2013
    Posts
    58

    What is a robots.txt file?

    What is a robots.txt file?

  2. #2
    Senior Member
    Join Date
    Jan 2013
    Posts
    138
    The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Creating a robots.txt file can actually improve your website indexation.

  3. #3
    Senior Member
    Join Date
    May 2013
    Posts
    198
    A robots.txt file is a text file that tells web crawler to crawling certain pages of website. The file is essentially a list of commands, such Allow and Disallow , that tell web crawlers which URLs they can or cannot retrieve.

  4. #4
    Registered User
    Join Date
    Apr 2014
    Posts
    51
    Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

  5. #5
    Member
    Join Date
    Oct 2014
    Posts
    32
    Robots.txt file is for Google bots, it helps to tell them, which webpage of your website need to be crawled and which should be avoided. It is always advisable to check the robots.txt of the website before any submission.

  6. #6
    Registered User
    Join Date
    Oct 2014
    Posts
    3
    Quote Originally Posted by akashram View Post
    A robots.txt file is a text file that tells web crawler to crawling certain pages of website. The file is essentially a list of commands, such Allow and Disallow , that tell web crawlers which URLs they can or cannot retrieve.
    Yes I agree with you,Its a file having the list of commands who tell the search engine crawler to crawling certain pages of website.
    Bruce Acacio(Chief Executive Officer) of INFINITE Corporation, and has played a crucial role in making the Orange County based software firm an international IT solutions provider.

  7. #7
    Registered User
    Join Date
    Aug 2014
    Posts
    658
    It is an HTML tag placed on the source of a web page which redirects search engine spiders which files to crawl on or not.
    Easily create a lending and borrowing script in a few days with Agriya's peer to peer lending and borrowing software - Crowdfunding Lend .

  8. #8
    Registered User
    Join Date
    Aug 2014
    Location
    13.524823 Langitute now you find Latitude
    Posts
    82
    Robot.txt file is a file through this file you can set the authorization for the search engine spider what they have to crawl and what not. if you are not allow the crawler to crawler your certain area of website than you can disallow robot via robot.txt.

  9. #9

  10. #10
    Registered User
    Join Date
    Jul 2014
    Location
    India
    Posts
    243
    A robots.txt document is a content record that stops web crawler programming, for example, Googlebot, from creeping certain pages of your website. The record is basically a rundown of summons, such Allow and Disallow , that tell web crawlers which Urls they can or can't recover.

  11. #11
    Senior Member
    Join Date
    Nov 2011
    Posts
    263
    With Robot.txt file you can block the web pages that you do not want bots to consider for crawling or indexing. If it is not made in a proper way, you may block the whole site from crawling which could create a big problem. It is always preferred to get it done by someone who has got a proper idea on how robot.txt is made. Or you can simply use robot.txt file file option in webmaster tools to know whether is is created properly or has got any issue.

  12. #12
    Registered User
    Join Date
    Dec 2012
    Location
    India
    Posts
    929
    Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

  13. #13
    Registered User Jackandrew's Avatar
    Join Date
    Jul 2014
    Location
    Los Angeles, Texas, United States
    Posts
    122
    The robots.txt file is a simple txt file that are placed on your server, if you go www.domain.com/robots.txt you see the file of websites that the site owner is asking the search engines to "skip" (or "disallow"). If any files and directories (which hurts your business) you don't want indexed by search engines, you can use a robots.txt file.

  14. #14
    Member
    Join Date
    Aug 2014
    Location
    India
    Posts
    11
    robots file has a .txt extension that contains instructions for crawlers whether to crawl the website and index its pages

  15. #15
    Registered User
    Join Date
    Oct 2014
    Location
    brighton
    Posts
    70
    Robots.txt file to control which pages of your site are indexed by search engines.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.