Results 1 to 11 of 11
  1. #1
    Member
    Join Date
    Sep 2011
    Posts
    81

    Why we use Robots.txt File ?

    anyone share your information?????

  2. #2
    Junior Member
    Join Date
    Sep 2011
    Location
    India
    Posts
    20
    robots.txt are also helpful in blocking the bots from indexing directories that contain scripts. If you have a very plain site that you would be ok with having the engines crawl everything than you do not need a robots.txt.

    as for xml sitemaps, just because the engines say they would like you to do something doesn't mean you should do it. for good webmasters i see no benefit from xml sitemaps. but we should discuss that in a seperate thread

    admin pages , paypal pages and other pages we can block here not to crawl these links.....

  3. #3
    Member
    Join Date
    Apr 2011
    Posts
    71
    Hello..
    Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders.it is an important factor for the webpages being indexed properly and very easy to setup.

  4. #4
    Registered User
    Join Date
    Sep 2011
    Posts
    388
    Robots.txt is a text (not html) file you put on your root directory to tell search robots which pages you would like them not to visit.
    The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it.

    If you use robots.txt with your Sitemap, all major search engines (Google, Yahoo, MSN) will discover it automatically.
    230% more traffic with 12+ Keyword Research Tools

  5. #5
    Registered User
    Join Date
    Sep 2011
    Posts
    65
    Robots.txt is a file that is used to exclude content from the crawling process of search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol. Robots.txt file usage is sometimes ignored. On the other hand, it is an important factor for the webpages being indexed properly and very easy to setup.
    Last edited by carolinehill; 05-16-2012 at 09:10 AM.

  6. #6
    Senior Member
    Join Date
    Feb 2007
    Posts
    294
    If you are willing to give permission to the search engines crawling robots to access your web server in full, with all images, folders and contents being accessible, there is actually no need to make use of Robots.txt file and you can find thousands of websites which do not contain the item. But the real point happens when you would like to block some parts to get accessed.

    For example, sometimes companies which have important information or several pictures folders on their servers, make use of Robots files to tell search engines not accessing the designated areas on their servers. Another example can be about shopping portals and naturally, a lot of their pages do not need to get indexed by search sites, so their admins block them through writing their URLs on the Robots files.

  7. #7
    Registered User
    Join Date
    Feb 2011
    Posts
    20
    Hi..

    Robots.txt is a text file you put on your site to tell search engine, which page you would like to not visit by them. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site. It is not a kind of password protection and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door.
    Here I would like to give an example like:you cannot prevent thieves from coming in but the good guys will not open to door and enter. Robots.txt is why we want to protect the some sensitive data from being indexed and displayed in search results.

    Thanks
    Search Engine Optimization NYC

  8. #8
    Registered User
    Join Date
    Oct 2011
    Posts
    1
    Robots.txt File is use for website crawling and indexing and You can tell Search Engines not to crawl your webpage through robots.txt

  9. #9
    we use this to hide some of the secret content on our site that we do not want to show visitors...

  10. #10
    Senior Member
    Join Date
    Oct 2011
    Location
    Delhi
    Posts
    159
    robot is called spider that spide your site and tell us which page you would like to spider or not

  11. #11
    Registered User
    Join Date
    Feb 2011
    Posts
    20
    Hello Friends..


    Robots.txt is a text file that you need to put on your site to tell search robots which pages of your website you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door.


    Thanks
    Internet marketing Newyork

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.