Results 1 to 4 of 4
  1. #1
    Senior Member
    Join Date
    May 2020
    Location
    Spain
    Posts
    982

    What is the limit of a robot.txt file?

    Hello friends,
    What is the limit of a robot.txt file?

  2. #2
    Senior Member
    Join Date
    Apr 2021
    Posts
    428
    Hello friends,

    Google highly respects the robots.txt file, but it is still a directive and not a mandate. File Size. Google supports a limit of 521 kilobytes for robots.txt files, and if the content exceeds this maximum size, it can ignore it. We don't know if other search engines also set a limit for these files. Robot txt. is cached 24 hours

  3. #3
    Senior Member
    Join Date
    Dec 2021
    Posts
    369
    500 kibibytes is what google says is the maximum for robots txt file. If you have pages you do not want indexed, you could use do not follow links and not put them in your sitemap

  4. #4
    Senior Member
    Join Date
    Sep 2021
    Location
    India
    Posts
    367
    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.