How to create robots.txt file..?

[swatijain2233]

Hello friends,

I want to know that,How to create robots.txt file..?

[arjupraja2]

There are many online software/tools are available on Internet, you just search on google "create robots.txt file".

[Jennifer Martin]

Creating Robots.txt file is very simple. You just need to open Notepad, write below code

User Agent: *
Allow: /

Sitemap:- (URL of your website Sitemap)

If you want to block any specific search engine from crawling your website, than you need to mention search engine name in front of user agent. Suppose I don't want my website to be crawled for Google Bots, then my robots file will be some what like this:-

User Agent: Google
Disallow: /

User Agent: *
Allow: /

This means other than Google bots all other crawlers can crawl the website.

If you want to Disallow specific folder or specific page of the website. You just need to add this line.
Disallow: /folder/
Disallow: /file.html
Disallow: /image.png

[othername0104]

The syntax for using the keywords is as follows:

User-agent: [the name of the robot the following rule applies to]

Disallow: [the URL path you want to block]

Allow: [the URL path in of a subdirectory, within a blocked parent directory, that you want to unblock]

[rajanvefgus]

You can easily create robots.txt file. make a robots.txt and use User-agent: *
Disallow: .

day picnic in delhi | Picnic near gurgaon

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[spindlow2016]

There are lot of tool available on the internet but we prefer to use the sitemap generator. It is a file which tell the search engine spider to which page has to be crawled or not by allowing or disallowing.

[Arangs]

The robots.txt file is a very simple format that can be created by the tool Notepad. If you use WordPress robots.txt file would look like this:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

“User-agent”:* which means that all search robots from Google, Yahoo and MSN should use this guide to find your site.
“Disallow: /wp-“: This line of code tells the search engines know it should not "rummaging" in the files of WordPress starts with wp-.

If you use Google Webmaster, they also allow you to create robots.txt

[tftus]

Simply add this code in notepad and save as robots.txt and upload it on your website's root.