Robots.txt?

[jesicawillss]

What is robots.txt? Is it a part of on-page?

[James-A]

It is a text file located in the root directory to restrict search engine spider's access to certain files or folders of a web site.

[vikaskatoch]

In simple language this file used to instruct to Google Robot than particular page will be index or no index and follow or no followed by Google Robot. If you instruct no indexed page than robot will not crawl the whole page.

[spinxwebdesign]

Yes it is a one part of on page. Robots.txt is a file that can be used to restrict search engine crawlers from various pages or sections of your site.

[davidharley8]

Robots.txt is a text file by name, has a special meaning for the majority of honorable robots on the web. By defining a few rules in this text file, you can tell the robots not to crawl and index certain files, folders on your site, or at least. For example, you do not want Google to crawl / images directory on your site because it is useless for you and wasting your bandwidth site. Robots.txt lets you tell Google just that.

[search4i]

Absolutely its a part of on page SEO. Robots.txt is the file where you can put the portion of your website that you want to hide from search engines to crawl.

[VijendraDiwakar]

Robot.txt is a text file located in the root directory.It is used to restrict certain files of a web site from search engine spider's access.

[mwaraitch]

its a file which restrict and allow various url of your website, you can limit the visibilty of your website or allow everything, the spider will act accordingly.

[dareckyoung]

Yeah! Its a On Page factor or part. It is text file. This is useful tool for your because it is helpful for crawling your site in search engine.

[colinwood07]

Yes !!!!

Robots.txt file is part of onpage & we set the permission for indexing & Crawling in search engine....

[swansmith]

u can use robot text file for follow or unfollow our website pages.

[pestindia86]

hi i am agree with your answer,thanks for share

[ajaykr86]

yes it is necessary for on page ..

[brainpulse]

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

[jesicawillss]

Thanks for the informative replies. Can you tell me that how can I make it practically in the website?

[JackiPhone]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

power by:- webconfs.com/what-is-robots-txt-article-12.php

[carolinehill]

There is a hidden, relentless force that permeates the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."

[wikklinks]

The robot.txt file determines which directories may be indexed. By default, Joomla 's robot.txt setting ,it is not allowed to index the images directory. For SEO purpose, change the robot.txt to allow index of images as images search is getting very popular today.

[newsky]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory (i.e. http://mydomain.com/robots.txt) and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.

The concept and structure of robots.txt has been developed more than a decade ago and if you are interested to learn more about it, visit http://www.robotstxt.org/ or you can go straight to the Standard for Robot Exclusion because in this article we will deal only with the most important aspects of a robots.txt file. Next we will continue with the structure a robots.txt file.