Page 1 of 2 12 LastLast
Results 1 to 15 of 16

Thread: robots.txt

  1. #1
    Member
    Join Date
    Oct 2009
    Posts
    53

    robots.txt

    What's the importance of robots.txt? How does it work?

  2. #2
    Guest Moderator ~ServerPoint~'s Avatar
    Join Date
    Nov 2007
    Posts
    1,846
    That is information for the robots which visit your web site. Like which page to index and which not etc
    I believe I have seen some articles regarding that somewhere in the net
    ServerPoint.com - a true hosting company offering online presence solutions since 1998
    Web Hosting, colocation, dedicated servers, Virtual Private Server (VPS) hosting
    Wholly owned multi homed network, servers and facilities

  3. #3
    Registered User
    Join Date
    Oct 2009
    Posts
    2
    Hi,

    Robots.txt file is mainly for search engines and robots.txt file can be used to prevent access to certain parts of a web site, if not correctly implemented, and it can also prevent access to the whole site.This may be possible that sometimes the robots.txt is the only reason for not indexing of website in many search engines.
    This file works mostly for all the search engines but there is one search engine searchspry where i didn't check this file.
    You can check from here http://www.searchspry.com

    Thanks.

  4. #4
    Registered User
    Join Date
    Oct 2009
    Posts
    2
    you can stop search engines to crawl certain pages of your website through Robots.txt

  5. #5
    Member
    Join Date
    Oct 2009
    Posts
    62
    is robots.txt and RSS is same? whats the difference between on both?
    Packing Supplies in United States of America

  6. #6
    Registered User
    Join Date
    Sep 2008
    Posts
    1

    How can make a Robots File & benifits

    robots.txt file in add sitemap link also becoz spider come on your site search a robots.txt firl and lookup a files instruction follow as a http://helplinelaw.com/ robots.txt .

    There are two important considerations when using /robots.txt:

    robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

    the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

  7. #7
    Junior Member
    Join Date
    Sep 2010
    Posts
    15
    Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it.
    SPINX - A DIGITAL AGENCY: A Leading Web Development Company in Los Angeles, offers innovative web solutions.
    www.spinxwebdesign.com

  8. #8

  9. #9
    Registered User
    Join Date
    Feb 2018
    Posts
    10
    It give command to the search engine which page user want to crawl and index from the website.

  10. #10
    Senior Member
    Join Date
    Nov 2015
    Location
    India
    Posts
    142
    Robots.txt is imprtortant for SEO. By using this, you can allow or disallow pages your website. Allow means Google bot can crawl the webpages and disallow means pages are not crawl by spiders.

  11. #11
    Registered User
    Join Date
    Jan 2013
    Posts
    734
    Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

  12. #12
    Registered User
    Join Date
    Jul 2018
    Location
    Essex, CT
    Posts
    1
    Thanks for sharing helpful information

  13. #13
    efore a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.

  14. #14
    Senior Member
    Join Date
    Jul 2018
    Location
    Chennai
    Posts
    311
    Google use rotots.txt to crawl or read you web page. We can make allow and disallow the search engine to crawl our web page through Robots.txt

  15. #15
    Member
    Join Date
    Jul 2018
    Location
    Delhi
    Posts
    90
    I think before asking such question you should Google these question first or do some R&D for the same.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.