Results 1 to 15 of 15
  1. #1
    Senior Member
    Join Date
    Feb 2015
    Location
    Bangalore
    Posts
    528

    Is there a limit to robots.txt file?

    Is there a limit to robots.txt file?

  2. #2
    Senior Member dennis123's Avatar
    Join Date
    Apr 2013
    Location
    Bangalore
    Posts
    3,627
    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.

  3. #3
    Registered User
    Join Date
    May 2015
    Location
    Surat, Gujarat.
    Posts
    362
    Robot file is the shot file that is only 500 KB. And its use is to allow or stop crawlers to crawl the website.

  4. #4
    Registered User 24x7servermanag's Avatar
    Join Date
    Jul 2017
    Location
    India
    Posts
    1,020
    That's correct! There is limit of 500 Kb for robot.txt file.
    Server Management Company
    India's Leading Managed Service Provider | Skype: techs24x7
    Cpanel Technical Discussions - Lets talk !

  5. #5
    Registered User friendhrm's Avatar
    Join Date
    Feb 2017
    Posts
    212
    I am think that there is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".

  6. #6
    Senior Member
    Join Date
    Jan 2017
    Location
    India
    Posts
    123
    As of my work experience in SEO there is no limit.
    Below mention website url I am going to share with you which is one of the longer ones that I've seen that's just being used to disallow spiders:

    http://www.whitehouse.gov/robots.txt

    I don't believe that I've ever seen a size limit listed anywhere.

  7. #7
    Registered User
    Join Date
    Jan 2016
    Posts
    66
    Google has a limit of only being able to process up to 500 KB present in the robots.txt file.

  8. #8
    Senior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    486
    Yes, it does have. now your doubt must be cleared. if you have any doubt left you can ask us or else mark the thread as solved/closed.


    Quote Originally Posted by bangalorewebgur View Post
    Is there a limit to robots.txt file?

  9. #9
    Member
    Join Date
    Mar 2018
    Location
    India
    Posts
    28
    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

  10. #10
    Senior Member
    Join Date
    Jun 2013
    Location
    Forum
    Posts
    5,019
    John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".
    Cheap VPS | $1 VPS Hosting
    Windows VPS Hosting | Windows with Remote Desktop
    Cheap Dedicated Server | Free IPMI Setup

  11. #11

  12. #12

  13. #13

  14. #14
    Senior Member
    Join Date
    Mar 2018
    Location
    Delhi
    Posts
    284
    Yes, the limit to robots.txt file is 500kb

  15. #15

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.