PDA

View Full Version : web crawlers and root directory



Alimoqi
03-03-2011, 11:27 PM
Hi everyone,
Does google index pages that are in the root directory on the server but not linked to from any of the pages on the website? I work with another guy on a few sites and we have some test pages in the root directory (so we can both access them) that are basically duplicates of some of the other pages and I don't want them crawled. Is this a problem or should I put them in a separate folder that is blocked by robots.txt?
Thanks!

markdev
03-28-2012, 08:16 AM
no. Big G Bot visits pages from one to another. if you havnt putted your link of that page anywhere on your site then Bot will not reach that page. I Belive that Bot always start with index files.

i have several other files like yahoo site validation file, bing files and some other files which has no relation with my public pages.If you using wordpress you can hide your perticular page from search engine visibility.