Is robots.txt file necessary

Hi
Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice
Thanks for your support

No it won't cause any problems and when the search engines access your website and find no robots files there, they will interpret this as your permission to crawl and index all the pages of your website.

It is not necessary to have robot.txt file. If you want to exclude some file and directory to protect your valuable information form malicious user then you can create a robot.txt file to inform search engine crawler to not crawler the specified web address.

Your current goals will give you understanding if you need that or not

Quote:

---

Originally Posted by esnc

Hi
but if i not put this file in the website, does this cause some problem, please advice

---

No, it's not going to cause any problem, if you don't going to put this file. However, it is required in case if you don't want some of the pages not to be indexed in SEs.

Also, the file can be used to indicate the crawler after how much days crawler should revisit the web site.

No, a robot.txt file is not needed or required. It can be helpful when used properly, but it can also cause problems when it is used and not configured properly.

Most of the time i have seen that large sites indicates the robots to crawl the posts only and not the comments and other things. Why is it so?

I think you should refer the guideline from google about robots.txt: http://www.google.com/support/webmas...n&answer=40360

Thanks bob, found that link very useful for robot.txt

I think that it is NOT necessary. I've optimized plenty of sites without robots.txt files and they ranked well in Google and Yahoo. I do however include an HTML sitemap -- which is just a basic page with a link to every page on my site on it.

It is not necessary, but if you want to block your web page than you have to create robots.txt file otherwise it's not necessary

I am agree with ebosysinida, if you want to block your web page than you have to create robots.txt file otherwise not

is there a way to find robot.txt syntax error?

If you don't put robots.txt file then there is no any single disadvantage of that. But SEO point of view if you put that file then it is beneficial for you.

I am sorry, i have no idea about it. You can browse to the internet for more details. I used to search only thru internet.

Ventrilo
Website Conversion

If you don't put robots.txt file then there is no any single disadvantage of that. But SEO point of view if you put that file then it is beneficial for you.

Really Robot.txt file is not necessary to the website. It is used to restrict the search engine bots to crowl any parts of the website.
If you not want any restriction to search engine crawling then no use of robot.txt