what is robotxt file?

what is robotxt file?

Robots.txt is a text file (no html) you put on your site to tell search robots which pages you don't want them to visit.By defining a few lines in this text file, you can instruct robots not crawling and indexing of certain files, folders within your site, or at all.

google crawls your sites with help of robotxt file. without it, your sites cannot crawl the page.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

robots.txt file is used to direct robot and tell the robots that which file you have to read and which you donot.

Quote:

---

Originally Posted by jasoncamroon

Robots.txt is a text file (no html) you put on your site to tell search robots which pages you don't want them to visit.By defining a few lines in this text file, you can instruct robots not crawling and indexing of certain files, folders within your site, or at all.

---

You make it so easy to understand about robots.txt

Great Job :)

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol

Robort.txt is text file which tells to search engine crawl webpages.

Robots.txt file is a text file that resides on your server, and controls a whole lot of features on your website. It’s a simple text file in which there are a few lines of text, but it’s very powerful that it can even decide whether your website should be shown on Google or not, what part of your website should be shown to the search engines.

Robotxt file is the file in the website which contains the details of accessable and non-accessable pages of the website crawlers. The search engine crawlers can only crawl into the accessable pages of the website. It enables the privacy of the website.

Robot.txt one type of text file and when search engine robot/spider enter of site then this text file pass the information to the search engine robot to leave these page with out crawling.

If you want to hide some pages of your website from search engine then you just need to mention all those pages into this text file and put it as disallow. so, by this way search engine spider will not crawl all those pages which you mention in this file. Also as search engine crawler first go to this file, you can also add your sitemap.xml url here. So, that you can increase the crawling of your website.

Robots.txt file utilization is sometimes ignored. However, it is most essential aspect for any websites being listed effectively & very easy to setup.

Robots.txt File is a regular text(not html)file that you put on your site. When a search engine crawler comes to your site. it will look for a special file on your site That file is called robots.txt File.

Hi,
i am agree with your answer