View Full Version : What is the limit of a robot.txt file?
ninadordev
02-16-2018, 01:27 AM
Hello friends,
What is the limit of a robot.txt file?
wellliving
02-16-2018, 01:53 AM
Hi
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb
https://www.welllivingshop.com/mattress-toppers/
Carlbrewster
02-16-2018, 01:54 AM
Despite of implementing Robots.txt, there are chances that the blocked page is crawled and indexed by Google.
veraajverma
02-16-2018, 01:59 AM
Google has a limit of only being able to process up to 500KB of your robots.txt file.
muslimastrolger
02-16-2018, 06:13 AM
500KB is the limits of robots file
easypeasystore
02-16-2018, 08:26 AM
Search Engine has a limit only 500KB of robots.txt file
smmexpertss
02-16-2018, 12:57 PM
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points
Powered by vBulletin® Version 4.2.4 Copyright © 2024 vBulletin Solutions, Inc. All rights reserved.