Page 1 of 2 12 LastLast
Results 1 to 15 of 18
  1. #1
    Senior Member
    Join Date
    Sep 2018
    Location
    Delhi
    Posts
    110

    Robots.txt files

    Hello folks

    I have little bit question regarding Robots.txt file

    Is it necessay to upload the robots.txt file for my website ?

    Please reply me asap...

  2. #2
    Registered User
    Join Date
    Jun 2018
    Posts
    1,416
    It's just an ordinary text file that you can then open with Notepad. If you don't have a robots.txt file in your site's root directory, then you can create one. All you need to do is create a new text file on your computer and save it as robots.txt. Then, just upload it to your site's root folder.

  3. #3
    Senior Member
    Join Date
    Jul 2018
    Posts
    454
    Robots.txt files control crawler access to certain areas of your site. While this can be very dangerous if you accidentally disallow Googlebot from crawling your entire site (!!), there are some situations in which a robots.txt file can be very handy.

    Some common use cases include:

    1. Preventing duplicate content from appearing in SERPs (note that meta robots is often a better choice for this)
    2. Keeping entire sections of a website private (for instance, your engineering team’s staging site)
    3. Keeping internal search results pages from showing up on a public SERP
    4. Specifying the location of the sitemap(s)
    5. Preventing search engines from indexing certain files on your website (images, PDFs, etc.)
    6. Specifying a crawl delay in order to prevent your servers from being overloaded when crawlers load multiple pieces of content at once

  4. #4
    yes, you should have robots.txt file

  5. #5

  6. #6
    Senior Member
    Join Date
    Sep 2018
    Location
    Delhi
    Posts
    110
    Quote Originally Posted by pharmasecure View Post
    It's just an ordinary text file that you can then open with Notepad. If you don't have a robots.txt file in your site's root directory, then you can create one. All you need to do is create a new text file on your computer and save it as robots.txt. Then, just upload it to your site's root folder.
    Ok..Thanks for sharing such kind of information with me. But, by default robots.txt file considered as allows ,then why it is necessary for my website

  7. #7
    Senior Member
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    5,019
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | Free IPMI Setup

  8. #8
    Junior Member
    Join Date
    Sep 2018
    Location
    Delhi
    Posts
    7
    Robot.txt is a text file that can be open with notepad. If you don,t have robot.txt file in your website then you should easily created.

  9. #9
    Registered User
    Join Date
    Sep 2017
    Posts
    259
    It's only a conventional content document that you would then be able to open with Notepad. On the off chance that you don't have a robots.txt record in your site's root index, at that point you can make one. You should simply make another content document on your PC and spare it as robots.txt. At that point, simply transfer it to your site's root organizer.

  10. #10
    Registered User
    Join Date
    Sep 2018
    Posts
    13
    The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file. Websites with sub-domains generally need a Robots.txt file for each sub-domain, all so that information that is not view able to the public is not picked for a keyword search.

  11. #11
    Registered User
    Join Date
    Aug 2017
    Location
    Mumbai, India
    Posts
    4
    It's not mandatory to have Robots.txt file but it a Good SEO practice. Robots.txt file is the first thing for which Google bots look after. As we mention which all search engines are allowed to crawl the website and if we have blocked any particular folder or pages from crawling.

  12. #12
    Registered User
    Join Date
    Nov 2017
    Location
    India
    Posts
    37
    When search engine crawlers come to your site it will be looking for some special file. i.e, call robots.txt file. This file tells to search engine spiders which pages of your site should be index and which pages of your site should be ignored.

  13. #13
    Registered User
    Join Date
    Jan 2018
    Posts
    546
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

  14. #14

  15. #15
    Registered User
    Join Date
    Aug 2018
    Location
    Atlanta, Georgia
    Posts
    62
    Hi,

    It's only a typical text document which you could start with Notepad. If you have a robots.txt file in your website's root directory, you then may cause one. All you have to do is make a new text document onto your own personal pc and store it as robots.txt.

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.