what is robotxt file?

[Kevin Farrell]

robots.txt is a simple text file which tell search engine what to index in website.

[Openxcell Inc]

Robots.txt is a text file which instruct crawler about the index and crawling of the pages of website. It is very useful for hiding some important pages of website.

[ebriks]

what is robotxt file?

Robots.txt is a simple text (not html) file you put on your website root directory to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all.

[john515]

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly.

[johnwilson639]

Robot.txt is a text file that tells the google robot ignore these pages from a websites.in a simple manner,if you want to uncrawl any page from search engine,then use a robot.txt file in front of that page.

[john12345]

Define Seo?

[john12345]

what is on page optimization?

[john12345]

Tell me about Penguin Update?

[john12345]

Define sitemap?

[john12345]

Tell me differnce between Blackhatseo and whitehatseo?

[ChrisDevlin]

In robots.txt file we can give command to all search engines bots or spiders, it helps to prevent web crawlers and other web robots from accessing all or part of a website. We can block any search engine bots by giving command in Robots.txt file.

[sabrinasai]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

[Raju]

Robot.txt file is the file used to hide the some content or page of our website from the spider.So that Spider not reach your personal data or privacy policy.Suppose you have a website and you don't want disclose privacy of a company then you use the robot.txt file.

[ajaykrgc]

hi,
It is called spider or crawler and say about web page i.e which page you would like to spide or not

[Brian-Alfaro]

Helps to protect data over web from Google "not to crawl that".Like JavaScript codes