View Full Version : How to generate robot.txt file on any website !
Lafaso870
03-09-2011, 10:46 PM
How to generate robot.txt file and suppose that you want block admin of any website then how to generate it.
please suggest me guys with such example.
gigsforgurus
03-10-2011, 04:04 AM
All you need to do is create a .txt file, name it robot.txt and drop it in to your root file in your ftp. In the robot.txt file, you can put:
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /
The Disallow: / means that any bots should not visit your site any further.
Hope that helps!
Click SSL
03-10-2011, 04:17 AM
Robots.txt example
User-agent: *
Disallow: /URL/
Disallow: /URL/
Disallow: /URL/
Abagail
03-10-2011, 05:21 AM
The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.
At its most simple, a robots.txt file looks like this:
User-agent: *
Disallow:
Powered by vBulletin® Version 4.2.4 Copyright © 2024 vBulletin Solutions, Inc. All rights reserved.