PDA

View Full Version : ROBOTS And Its applications



ajaykrgc
02-15-2012, 12:08 AM
hi,
what is robots.txt file and its code for block all page in a site ?

Amy.Sarin
02-20-2012, 12:29 AM
Robots.txt is a text (not html) file you put on your root directory to tell spiders which files to ignore (or alternatively) which files to crawl.

It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.

I wonder why you want to block all the pages. The whole purpose of optimizing your website gets defeated.
You should place this in your robots.txt file,
User-agent: *
Disallow: /

You may use noindex tags as well.