Results 1 to 10 of 10
  1. #1
    Senior Member
    Join Date
    Sep 2015
    Posts
    251

    What is the limit of a robot.txt file?

    Hello friends,

    What is the limit of a robot.txt file?

  2. #2
    Registered User 24x7servermanag's Avatar
    Join Date
    Jul 2017
    Location
    India
    Posts
    1,020
    There is limit of 500KB of the robot.txt file. You can ask your web hosting provider to upload it under your control panel (root directory of the website) and webmaster will pick it automatically.

    If you have access then you can upload it from your end.
    Server Management Company
    India's Leading Managed Service Provider | Skype: techs24x7
    Cpanel Technical Discussions - Lets talk !

  3. #3
    Member
    Join Date
    Jan 2018
    Posts
    66
    I am think that there is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".

  4. #4
    Senior Member deepakrajput's Avatar
    Join Date
    Feb 2012
    Location
    CA
    Posts
    1,223
    Robots.txt file is simply a notepad, including some set of instructions.

  5. #5
    Senior Member
    Join Date
    May 2017
    Posts
    605
    Robots.txt max size is 500KB. As stated on Google's Robots.txt Specifications page, “A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500 kilobytes (KB).”

  6. #6
    Senior Member
    Join Date
    Sep 2017
    Location
    New york
    Posts
    117
    Yes, there is a limit to robots.txt file in terms of filesize 500 KB

  7. #7
    Registered User
    Join Date
    Jan 2018
    Posts
    5

    Robots.txt

    Quote Originally Posted by parmodshastri View Post
    Yes, there is a limit to robots.txt file in terms of filesize 500 KB
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.

  8. #8

  9. #9
    Registered User
    Join Date
    Feb 2016
    Location
    Bangalore
    Posts
    696
    I think google has a limit only 500Kb of your robots.txt file

  10. #10
    Registered User
    Join Date
    Aug 2017
    Location
    United States
    Posts
    148
    I think that there is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".
    Source- Google: Robots.txt Files Must Be Smaller Than 500KB

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.