PDA

View Full Version : what is robot text?



sonuS
08-08-2017, 01:53 PM
what is robot text?

HiteshPatel
08-09-2017, 12:51 AM
it gives instruction to search engine spider that which pages of your sites you want to index and which pages want to ignore, there are many pages in website but not all useful for users so it's not important to index those pages in search engine, A robots.txt file tell to the robots which pages or parts of the website that should be index and which parts should not.

an example of Robots.txt file is below:
(!) User-agent: Googlebot/MSNbot/yahoobot.
Disallow: add your link here which do not want to index.

(!!) User-agent:*
Disallow: /

(!!!)(*) indicates all the search engines.
(/) indicates all pages of a site.

bmarlowecarol
08-09-2017, 01:07 AM
Hi I am Bmarl from United States and I work with one of the most reputed company named - Ancientwriters who provide the perfect writing as well as publishing services . I have been working there from past few years. If you are looking for such writing and publishing services then you can visit their website.

Jacksondavid
08-09-2017, 01:10 AM
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

meera64
08-09-2017, 04:59 AM
robots.txt is a text file its helps to block all web crawler from all web pages or specific web crawler from specific web pages.

BenWood
08-09-2017, 06:48 AM
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

daikaads
08-09-2017, 09:52 AM
Robots.txt is a file which is used by webmasters to instruct search engine robots not to crawl the web pages in the website.

deepakrajput
08-09-2017, 11:45 PM
Robots.txt file help to control your site indexing by Google, Bing & Yahoo.

jaysh4922
08-14-2017, 07:58 AM
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

jonny deep
08-14-2017, 09:10 AM
Robots.txt is a special kind of file that is not HTML or any other type. The purpose of having a robots.txt file is to tell the search engine that what content on your site is allowing search bots and indexes.

leena13
08-16-2017, 01:29 AM
robots.txt is text file. it helps to block all web crawler from all pages or specific web crawler form specific web page.

neelseofast
08-16-2017, 01:44 AM
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

lavyhair
08-16-2017, 04:13 AM
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

wellliving
08-16-2017, 05:36 AM
It is Robot txt and this means that there are certain places on the website where there is personal information and in these places customers are not allowed to go so in this case robot txt is used

24x7servermanag
08-16-2017, 07:13 AM
You can refer it - http://forums.hostsearch.com/showthread.php?99005-What-is-Robots-txt :)