PDA

View Full Version : What is robot.txt?



danialrobart
09-08-2014, 08:47 AM
What is robot.txt?
Manchester escort (http://www.escortservicesmanchester.co.uk)

robingage
09-08-2014, 09:11 AM
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.

JessicaJohn
09-08-2014, 09:20 AM
It is a text file which you use to guide search engine robots that which pages or links you don't want to allow, it is very important file according to SEO point of view.

ryanmax
09-08-2014, 09:32 AM
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

jaysh4922
09-09-2014, 03:50 AM
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

ShaliniVerma
09-09-2014, 04:04 AM
Robot.txt is a way to inform search engines which page to follow and rank. If we dont have a robots.txt file google or any other search engine will by default index & follow each page on our website.

e.g, facebook blocks image section of every user to get followed & indexed by google try facebook.com/robots.txt for better idea.

kendy1402
09-09-2014, 05:23 AM
Robots.txt is file controls to block or allow the operation of the bot is looking into your website

joshaidan
09-09-2014, 06:41 AM
Robot.txt - Web page not indexed in Bot database.

evajasmine
09-09-2014, 07:06 AM
Robot.txt is the text file that tells search engine bot to understand what should be crawled and what shouldn't be crawl from that search engine bot itself. you can protect your private pages from indexing by using disallow property in robots.txt file.

nainasharma1
09-09-2014, 07:22 AM
Robots.txt file is txt file. It inform SE about the pages of sites.

Rajdeep Bose
09-11-2014, 02:02 AM
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

ChristinaCa
09-11-2014, 03:02 AM
The Robots.txt is a text documents file that is informs search engine spider which parts of your website be Crawled and which parts you do not want the search engines Crawling.

rajivdevkk
09-11-2014, 03:36 AM
Robots.txt is used to crawls the pages and block the pages. if you don't want to crwals in search engine. you can block it by using robots.txt file.

User-agent: *
disllow: /dafafa


day picnic around delhi (http://www.arounddelhi.net/?page_id=203) | day picnic near delhi (http://www.conventionindia.in/Daysout.aspx)

nilam-shah
11-10-2014, 07:21 AM
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site...........