Hello friends,
What is the limit of a robot.txt file?
Results 1 to 7 of 7
-
02-16-2018, 01:27 AM #1Registered User
- Join Date
- Dec 2016
- Location
- India
- Posts
- 143
What is the limit of a robot.txt file?
-
02-16-2018, 01:53 AM #2Registered User
- Join Date
- Dec 2015
- Location
- USA
- Posts
- 1,060
Hi
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb
https://www.welllivingshop.com/mattress-toppers/
-
02-16-2018, 01:54 AM #3Registered User
- Join Date
- Aug 2016
- Posts
- 182
Despite of implementing Robots.txt, there are chances that the blocked page is crawled and indexed by Google.
-
02-16-2018, 01:59 AM #4Registered User
- Join Date
- Mar 2016
- Posts
- 294
Google has a limit of only being able to process up to 500KB of your robots.txt file.
Medicine Delivery App | Book Diagnostic Lab | Buy Medicines Online | Buy Patanjali Products Online | Buy Vaccines Online | Online Medical Store | HerCare | Periods and Fertility Tracker | Menstrual Cycle Tracker | Buy Beauty Care Products Online | Buy Baby Care Products Online | Buy Health Supplements Online
-
02-16-2018, 06:13 AM #5Member
- Join Date
- Aug 2015
- Location
- jaipur
- Posts
- 30
500KB is the limits of robots file
-
02-16-2018, 08:26 AM #6Registered User
- Join Date
- Feb 2018
- Location
- United Kingdom
- Posts
- 11
Search Engine has a limit only 500KB of robots.txt file
-
02-16-2018, 12:57 PM #7Senior Member
- Join Date
- Nov 2015
- Location
- United Kingdom
- Posts
- 241
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points
Reply With Quote
Posting Permissions
|
Bookmarks