Results 1 to 4 of 4
  1. #1
    Senior Member
    Join Date
    May 2020
    Location
    Spain
    Posts
    982

    What is the limit of a robot.txt file?

    Hello friends,
    What is the limit of a robot.txt file?

  2. #2
    Senior Member
    Join Date
    Apr 2021
    Posts
    614
    Hello friends,

    Google highly respects the robots.txt file, but it is still a directive and not a mandate. File Size. Google supports a limit of 521 kilobytes for robots.txt files, and if the content exceeds this maximum size, it can ignore it. We don't know if other search engines also set a limit for these files. Robot txt. is cached 24 hours

  3. #3

  4. #4
    Senior Member
    Join Date
    Sep 2021
    Location
    India
    Posts
    464
    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.