Webcrawlers can be prevented from accessing certain directories of your website by using the disallow option in your robot.txt file.
Web site owners use the /robots.txt file to give instructions about their site to web robots. It works likes this: A robot wants to vist a website URL (example:
http://www.example.com/welcome.html). Before it does, it checks for
http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. If you need to prevent the robots from accessing the cgi-bin directory, use the following lines in your robot.txt file:
User-agent: *
Disallow: /cgi-bin/
Robots directives for Disallow/Allow are case-sensitive. Use the correct capitalization to match your website.
Bookmarks