Robots.txt

[itznehamali]

What is Robots.txt file?

[nancyisabell]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

[rohitnagpure336]

Robots.txt files are Important files for website..? Can we must need to keep this..

[cpsi]

Robots.txt is very important for crawling your website.The robots exclusion standard, also known as the robots exclusion protocol or robots.txt protocol, is a standard used by websites to communicate with web crawlers and other web robots.

[edwina]

Nice information.

[raheel]

Great described.

[rajivdevkk]

Robots.txt is the txt files which helps to crawls or don't crawls the site pages. For Example, if you want your site pages don't crawls, you can use robots.txt file.

User-agent: *
Disallow: /backend/


day picnic spots in delhi (http://www.arounddelhi.net/?page_id=203) | picnic spots in delhi ncr (http://www.conventionindia.in/Daysout.aspx)

[Tausifhmp]

Robots.txt is used to inform google about which web page of your website you should be crawled and which should not be

[daviddakarai]

The robots.txt file defines how a search engine spider like Googlebot should interact with the pages and files of your web site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. The robots.txt is a very simple text file placed on your web server.

[RH-Calvin]

It is a text file used to define the functions of search engine crawlers. They contain instructions to crawl and disallow certain webpages in your website.

[jane1]

The robot.txt file is used when a website owner or webmaster likes that search engine robots do not visit their special webpage.

[sonuS]

It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage.

[ramskl]

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

[rizwanaegis1942]

You can block your webpages and folders to get crawl and indexing in any search engine.

[kiernprice]

Robots.txt protocol, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies the instruction format to be used to inform the robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.

[anirban09P]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.