Results 1 to 13 of 13

Thread: Robot.txt File

  1. #1
    Senior Member
    Join Date
    Sep 2018
    Location
    Delhi
    Posts
    110

    Robot.txt File

    Hello Friends

    Can anyone tell me where to submit robot.txt file in our website ???

    Please tell me

  2. #2
    Registered User georgeb's Avatar
    Join Date
    Apr 2012
    Location
    Montreal, CA
    Posts
    18
    Please look here.

  3. #3
    Registered User
    Join Date
    Jul 2016
    Location
    India
    Posts
    37
    In the Root folder of your site. Make sure you have used correct syntax to block those segment of site which are of no use.

    Thanks

  4. #4
    Registered User
    Join Date
    Jun 2018
    Posts
    1,416
    Keep in mind that only one file or folder can be used per Disallow line. You may add as many Disallow lines as you need. Once complete, save and upload your robots.txt file to the root directory of your site. For example, if your domain is www.mydomain.com, you will place the file at www.mydomain.com/robots.txt.

  5. #5
    Senior Member
    Join Date
    Jul 2018
    Posts
    454
    Simply type in your root domain, then add /robots.txt to the end of the URL. For instance, Moz's robots file is located at moz.com/robots.txt.

  6. #6
    Registered User
    Join Date
    Sep 2017
    Posts
    259
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

  7. #7
    Registered User
    Join Date
    Jan 2013
    Posts
    734
    Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

  8. #8
    Registered User
    Join Date
    Nov 2017
    Location
    India
    Posts
    37
    hen search engine crawlers come to your site it will be looking for some special file. i.e, call robots.txt file. This file tells to search engine spiders which pages of your site should be index and which pages of your site should be ignored.

  9. #9
    Registered User
    Join Date
    Jan 2018
    Posts
    546
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

  10. #10
    Senior Member
    Join Date
    Aug 2017
    Location
    8th Floor, Olympia National Towers, Block 3, A3 & A4, North Phase, Guindy Industrial Estate, Chennai - 600032, Tamilnadu, India.
    Posts
    486
    robots.txt file is giving instructions to the Crawler or Search Engine Spider to Crawl and Block Particular Website URLs.


    For your information:

    Prateektechnosoft is a Netsuite Partner and expertise in NetSuite ERP, CRM, Cloud CRM, PSA and other Netsuite Solutions. And also providing Netsuite services of implementation, integration, support & development services.

  11. #11
    Senior Member
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    5,019
    Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
    Cheap VPS | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | Free IPMI Setup

  12. #12
    Registered User
    Join Date
    Feb 2016
    Location
    Bangalore
    Posts
    696
    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.

  13. #13
    Senior Member
    Join Date
    Feb 2016
    Location
    Bangladesh
    Posts
    486
    Robots.txt is a text file that contain instructions for search engine spiders. The files lists pages to allow and disallow Google crawling. Robots.txt file helps to control the crawling of your website.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.