PDA

View Full Version : How to add robots.txt?



PeteMrcus90
01-02-2016, 05:15 AM
Hello all, how to add robots.txt to my website http://www.enerco.co.uk
Is it necessary to add robots.txt in my website?
Please suggest me, thanks

Joylinford
01-02-2016, 11:51 AM
Robots.txt file is placed at root folder of domain. It would be like below.

User agent: *

Allow:/services

disallow: /admin

dennis123
02-01-2016, 05:54 AM
You can create robots.txt file by webmaster tools or create manually by opening notepad & create text file, then save as robots.txt and upload it in server.

towertech
02-01-2016, 06:13 AM
Hi,

Yeah It;s necessary to having robots.txt file on your website.

Open notepade and put the following text on it

User-agent: *
Disallow:

now save it with robots name.

Now you need to upload that robots.txt file on root directory.

Thanks

RH-Calvin
02-01-2016, 06:43 AM
Robots.txt is a text file that is defined in your website. It helps to contain instructions for crawlers to crawl and not to crawl certain webpages that are listed in the text file.

healthcare
02-02-2016, 02:15 AM
Hello Everyone

Keep in mind that only one file or folder can be used per Disallow line. You may add as many Disallow lines as you need. Once complete, save and upload your robots.txt file to the root directory of your site. For example, if your domain is www.mydomainname, you will place the file at www.mydomainname.com/robots.txt.

rajanvefgus
02-02-2016, 02:26 AM
Robots.txt is file and it's very helpful for to crawl the site page or don't crawls the site page. If you have many site pages, you should use robots.txt file.

day picnic spot delhi (http://www.arounddelhi.net/?page_id=203) | picnic spots delhi (http://www.conventionindia.in/Daysout.aspx)

Rajdeep Bose
02-02-2016, 02:30 AM
Robots.txt is the text file that is mostly used to instruct search engine which page should be crawled and which shouldn't be crawled.

Brucey
02-02-2016, 04:19 AM
Add it in the host folder

rahul3214
04-08-2016, 01:24 AM
First open notepad and type these

User-agent : *
Disallow: /
then save by name robots.txt file where all files saved.
Then put your domain name www.yourdomain.com/robots.txt\
and see the file

Put your URL in Disallow and allow to crawl or not crawl your page