Page 1 of 3 123 LastLast
Results 1 to 15 of 39
  1. #1
    Registered User
    Join Date
    Mar 2012
    Posts
    102

    What are Robots in SEO ?

    Hello friends
    Please give me the answer.

  2. #2
    Registered User
    Join Date
    Jun 2012
    Posts
    63
    I think this file name that controls the crawler to visit your that page of website which is under programing

  3. #3
    Member
    Join Date
    Sep 2011
    Posts
    82
    Robots.txt is file which used to restrict website visit to search engine crawler.

  4. #4
    Senior Member
    Join Date
    Aug 2012
    Posts
    284
    Robots are search engine crawler. They are useful for fetching the data from the website.

  5. #5
    Registered User
    Join Date
    Sep 2012
    Posts
    39
    Robots are search engine spider that crawls and indexes web pages.

  6. #6
    Registered User
    Join Date
    Sep 2012
    Location
    United State
    Posts
    10
    Robots is a catch-all, or generic term for programs and automated scripts that “crawl” through the web (the Internet) and collect data from websites and anything else on the Internet that they can find.

  7. #7
    Senior Member
    Join Date
    Apr 2012
    Posts
    1,019
    I think Robot.txt is used to prevent crawlers crawl on a certain page...

  8. #8
    Registered User
    Join Date
    Jun 2012
    Posts
    152
    Robots is illegal and it's search called search engine spider.

  9. #9
    Member
    Join Date
    Jul 2012
    Posts
    53
    Very useful information..thanks

  10. #10
    Registered User
    Join Date
    Sep 2012
    Posts
    19
    when a particular site is placed inside robots.txt the site is nor crawled by the google search engine.

  11. #11
    Registered User Engagedots's Avatar
    Join Date
    Sep 2012
    Location
    India
    Posts
    97
    It is to tell search robots which pages you would like them not to visit.

  12. #12
    Junior Member
    Join Date
    Jul 2012
    Location
    UK
    Posts
    6
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sensitive data, it is too naive to rely on robots.txt to protect it from being indexed and displayed in search results.

  13. #13
    Junior Member
    Join Date
    Oct 2012
    Posts
    14
    Three-link way is a process that is used for online site optimization as it can widen the scope of links that can be directed to your own page.

  14. #14
    Senior Member
    Join Date
    Apr 2019
    Location
    Mumbai
    Posts
    355
    Quote Originally Posted by Harry Mark View Post
    Hello friends
    Please give me the answer.
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.

  15. #15
    Senior Member
    Join Date
    Nov 2018
    Posts
    1,853
    Robots.txt is a file that give instructions to the crawler as to which page to crawl and which page not to crawl.

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.