Is robots.txt file necessary

[esnc]

Hi
Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice
Thanks for your support

[bermuda]

No it won't cause any problems and when the search engines access your website and find no robots files there, they will interpret this as your permission to crawl and index all the pages of your website.

[Robdale]

It is not necessary to have robot.txt file. If you want to exclude some file and directory to protect your valuable information form malicious user then you can create a robot.txt file to inform search engine crawler to not crawler the specified web address.

[~ServerPoint~]

Your current goals will give you understanding if you need that or not

[shennon]

Hi
but if i not put this file in the website, does this cause some problem, please advice

No, it's not going to cause any problem, if you don't going to put this file. However, it is required in case if you don't want some of the pages not to be indexed in SEs.

[Robdale]

Also, the file can be used to indicate the crawler after how much days crawler should revisit the web site.

[sasakakee]

No, a robot.txt file is not needed or required. It can be helpful when used properly, but it can also cause problems when it is used and not configured properly.

[Robdale]

Most of the time i have seen that large sites indicates the robots to crawl the posts only and not the comments and other things. Why is it so?

[bob]

I think you should refer the guideline from google about robots.txt: http://www.google.com/support/webmas...n&answer=40360

[Robdale]

Thanks bob, found that link very useful for robot.txt

[Oracle]

I think that it is NOT necessary. I've optimized plenty of sites without robots.txt files and they ranked well in Google and Yahoo. I do however include an HTML sitemap -- which is just a basic page with a link to every page on my site on it.

[ebosysindia]

It is not necessary, but if you want to block your web page than you have to create robots.txt file otherwise it's not necessary

[stevepeterson]

I am agree with ebosysinida, if you want to block your web page than you have to create robots.txt file otherwise not

[Jenie0109]

is there a way to find robot.txt syntax error?

[spinxwebdesign]

If you don't put robots.txt file then there is no any single disadvantage of that. But SEO point of view if you put that file then it is beneficial for you.