Log in

View Full Version : SEs Check robots.txt and just go away?



pooja escorts
09-20-2021, 10:11 PM
Help, guys! I submitted my site a month ago. I just checked my logs, and I can see that
there have been 40 requests for robots.txt file, where they check that file, and nothing else.
After checking the file, they go away. Is this normal? Do search engines just check it, then come back later to crawl?
Here's my robots.txt file:

User-agent: *
Disallow: /hid/
Disallow: /images/
Allow: /
Disallow:

Anything wrong with this?

Note:
The "Allow" is there to invite the engines to crawl everything.
Anybody know if this works?

Sopie1805
09-21-2021, 05:11 AM
The "allow" should not be there, there is no "allow" directive for the robots.txt file. Likely they are ignoring the "allow" and interpreting the "/" (root) as disallow everything.
I would change it to just read:

User-agent: *
Disallow: /hid/
Disallow: /images/
Allow: /
Disallow: