PDA

View Full Version : What is the Important of robot.txt for a site in SEO?



Mikeleonard
12-22-2011, 02:48 AM
What is the Important of robot.txt for a site in SEO?

Ken Peterson
12-22-2011, 07:16 AM
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

Marksteve
12-22-2011, 07:51 AM
It is an important factor for a site to be indexed properly. it is not guaranteed that all search engines visit robot.txt and function accordingly. If it is configured to ignore, then certainly robot.txt is of no use.

ashish123
12-22-2011, 08:07 AM
robot.txt is used in website .this is help for all up data information collection

mesasa
12-22-2011, 08:13 AM
In short, if you want to hide some the pages or whole directory from search engine, you can disallow such pages in robots.txt file.
For example, You can block your admin panel URLs.

amitmax
12-22-2011, 10:01 AM
It helps to save website from SE penalty if using duplicate content on different pages.We can strict SEs to visit those pages by robot.txt file.
Thanks

onlinestore
12-23-2011, 01:14 AM
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search.

poul111
12-23-2011, 02:14 AM
It is an essential aspect for a site to be listed effectively. it is not confirmed that all google check out software.txt and operate accordingly. If it is designed to neglect, then certainly software.txt is of no use.

Abbott123
12-26-2011, 08:09 AM
It is an essential aspect for a site to be listed effectively. it is not confirmed that all google check out software.

jack12
12-26-2011, 09:03 AM
When look for engines optimization crawler comes to your website, it will look for a special submit on your website. That submit is called robots.txt and it tells google optimization spider, which Websites of your website should be indexed and which Websites should be ignored.

johnnina82
12-26-2011, 10:37 AM
When look for engines optimization crawler comes to your website, it will look for a special submit on your website. That submit is called robots.txt and it tells google optimization spider, which Websites of your website should be indexed and which Websites should be ignored.

very informative for me.

onlinestore
12-31-2011, 02:09 AM
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

onlineshop
02-28-2012, 06:01 AM
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.