How to generate robot.txt file and suppose that you want block admin of any website then how to generate it.
please suggest me guys with such example.
Results 1 to 4 of 4
-
03-09-2011, 10:46 PM #1Junior Member
- Join Date
- Jan 2011
- Posts
- 22
How to generate robot.txt file on any website !
-
03-10-2011, 04:04 AM #2Junior Member
- Join Date
- Mar 2011
- Posts
- 8
All you need to do is create a .txt file, name it robot.txt and drop it in to your root file in your ftp. In the robot.txt file, you can put:
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /
The Disallow: / means that any bots should not visit your site any further.
Hope that helps!Gigs For Gurus
Fiverr For Internet Marketers
Save 25% on HostGator hosting with coupon code: HG25COUPON
Source: Newest HostGator Coupon
-
03-10-2011, 04:17 AM #3Registered User
- Join Date
- Mar 2010
- Location
- Newark, DE, USA
- Posts
- 301
Robots.txt example
User-agent: *
Disallow: /URL/
Disallow: /URL/
Disallow: /URL/
-
03-10-2011, 05:21 AM #4Member
- Join Date
- Mar 2011
- Location
- New Delhi
- Posts
- 59
The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.
At its most simple, a robots.txt file looks like this:
User-agent: *
Disallow:
Reply With Quote
Posting Permissions
|
Bookmarks