PDA

View Full Version : What is the limit of a robot.txt file?



makoo
01-12-2022, 06:00 AM
Hello friends,
What is the limit of a robot.txt file?

Electrum
01-12-2022, 07:34 AM
Hello friends,

Google highly respects the robots.txt file, but it is still a directive and not a mandate. File Size. Google supports a limit of 521 kilobytes for robots.txt files, and if the content exceeds this maximum size, it can ignore it. We don't know if other search engines also set a limit for these files. Robot txt. is cached 24 hours

chris26
01-12-2022, 08:56 AM
500 kibibytes is what google says is the maximum for robots txt file. If you have pages you do not want indexed, you could use do not follow links and not put them in your sitemap

Cyril30
01-12-2022, 10:52 PM
Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.