PDA

View Full Version : Is your website really needs robots.txt?



denniemark
09-04-2012, 04:09 AM
We all know that before Google bot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. But Google webmasters suggest that if you want search engines to index everything in your site. You don't need a robots.txt file. But still lots of website webmaster use robots.txt, while they want search engines to index everything in your site. Google never suggest making a robots.txt file for every website, but webmasters creates.

ankita123
09-04-2012, 06:01 AM
Of course it need to seo.

Venus Brown
09-04-2012, 06:03 AM
People use robot file to block certain page from crawling like those "under construction." If you want to make all the pages crawlable , then there is no need to make it.

smitholivia
09-04-2012, 08:17 AM
where did you read google do not suggest to create robots.txt file.Matt cutts suggest website should have robots.txt file.

maxjohn148
09-04-2012, 08:44 AM
According to seo point of view robots.txt file should be there.

rehan12
09-04-2012, 05:51 PM
well as per latest updates no more use with them!!!!!!!!!!!!

amittrch
09-12-2012, 06:15 AM
Hi,
My website really needs robot.txt file that tell us which page you would like to spider or not.

Paul Jameson
09-12-2012, 09:35 AM
Robot.txt helps your website to suggest pages to crawl search engine. By SEO point of view Robot.txt needs.

razdev
09-12-2012, 10:35 AM
Hi,
if you want to your website will or not each page then need robots.txt file

mikeslough5
09-13-2012, 05:43 AM
Robot text file will use when you don't want that crawler don't see you specific page of your site.

borgninestallon
09-13-2012, 06:23 AM
If your site is very well structured and if you don't want search engines not to crawl any of your site pages, robot.txt will not be required for your site.

I never use robot.txt in my site which i am optimizing.

denniemark
09-25-2012, 08:50 AM
Robot.txt helps your website to suggest pages to crawl search engine.

It,s is wrong. Suppose your website is not have robots.txt file, then search engine will not crawl the website?

denniemark
10-05-2012, 07:31 AM
where did you read google do not suggest to create robots.txt file.Matt cutts suggest website should have robots.txt file.

Read this Goolge Post (Read the 3rd paragraph from the Top): http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

abhishekabhi
10-06-2012, 07:56 AM
HI


Yes every website need robot .txt for seo purpose.

abhishekabhi
10-06-2012, 07:58 AM
Can you know pest control?????

Charls007
10-06-2012, 08:14 AM
Yes we needed robot.text for crawling of my website page.

denniemark
10-19-2012, 04:32 AM
HI


Yes every website need robot .txt for seo purpose.

you mean. if a website not have a robots.txt file. then you can not SEO of that site

jennyhelzer90
10-19-2012, 05:19 AM
Robots.txt file on a website will function as a request that specified robots, ignore specified files or directories when crawling a site. So it is very important.