View Full Version : What is the limit of a robot.txt file?

05-01-2019, 03:47 AM
Hello friends,

I Want to Know that What is the limit of a robot.txt file?

05-01-2019, 03:56 AM
there's no limit for robots.txt

05-01-2019, 06:16 AM
Googlebot will only read the first 500kb. If your robots.txt is longer, it can result in a line being truncated in an unwanted way. The simple solution is to limit your robots.txt files to a reasonable size.

05-01-2019, 09:51 AM
Your robots.txt file must be smaller than 500KB

05-02-2019, 05:57 AM
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

05-02-2019, 06:03 AM
Google has apply the siz limit to robot.txt file with filesize 500kb .Content after the 500kb may be ignored.

05-02-2019, 06:04 AM
Google has a limit of only being able to process up to 500KB of your robots.txt file.

05-02-2019, 08:45 AM
You cannot make search engine ignore separate directories which have private files.