PDA

View Full Version : How to generate robot.txt file on any website !



Lafaso870
03-09-2011, 10:46 PM
How to generate robot.txt file and suppose that you want block admin of any website then how to generate it.
please suggest me guys with such example.

gigsforgurus
03-10-2011, 04:04 AM
All you need to do is create a .txt file, name it robot.txt and drop it in to your root file in your ftp. In the robot.txt file, you can put:

# robots.txt for http://www.example.com/
User-agent: *
Disallow: /

The Disallow: / means that any bots should not visit your site any further.

Hope that helps!

Click SSL
03-10-2011, 04:17 AM
Robots.txt example

User-agent: *
Disallow: /URL/
Disallow: /URL/
Disallow: /URL/

Abagail
03-10-2011, 05:21 AM
The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.

At its most simple, a robots.txt file looks like this:

User-agent: *
Disallow: