PDA

View Full Version : What Is A Robots.txt File?



ThedorisJackson
07-26-2014, 07:57 AM
Hi Friend,
Please can tell me about something about robots.txt file?

terrymoore
07-26-2014, 09:02 AM
The content of this file tells search engine crawlers how they should visit your site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go.

drwebsites
07-26-2014, 04:14 PM
A robots.txt file is a text file that stops web crawler software, such as Googlebot, from crawling certain pages of your site. If a page is blocked in your robots.txt, it will not appear in Google Search results.

stevex
07-28-2014, 07:11 AM
This file give the instructions to the robots about the site for crawling.

AvniShergill
07-28-2014, 07:19 AM
Hi Friend,
Please can tell me about something about robots.txt file?

Robots.txt file tells crawlers, which page of your website need to be crawled and which need to be avoided.

FrederickKopp
07-29-2014, 06:56 AM
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means obligatory for search engines but generally search engines obey what they are asked not to do.

onlinebigboss
07-29-2014, 08:10 AM
if you want Google bot not crawled your page in your sites so this condition you can also apply robot.txt file

Bestseoservices
07-29-2014, 08:20 AM
Robot.txt is a written text (not html) computer file you put on your website to tell look for robots which webpages you would like them not to check out.

User-agent: *
Disallow: /folder/

uthra
07-31-2014, 08:36 AM
If your mentioned the particular URL of your website in your robots.txt file,Google would not crawl that particular page and URL. That is the working process of robots.txt file.

jaysh4922
08-01-2014, 02:01 AM
Robots.txt is frequent name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to have the funds for directions about the Web site to Web robots and spiders.

customerdirect
08-01-2014, 03:21 AM
thats the good answer jaysh4922

sharmalve
08-01-2014, 06:31 AM
In simple language if you want to don't crawl the page by Google Bot search engine using robot.txt file.

Ezio
08-02-2014, 01:28 AM
Robots.txt is a text file webmasters create to instruct robots specially for search engine robots how to crawl and index pages on their website.

Ship497
08-02-2014, 01:59 AM
Its a .txt file which tells the search engine bots which pages on a website they have to crawl for indexing.

elislayal
08-02-2014, 04:11 AM
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a website.

peterdrucker27
08-02-2014, 08:52 AM
Robot.txt is a written text computer file you put on your website to tell google robots which webpages you would like them not to crawl out.

marclewis
08-04-2014, 03:05 AM
In short, the file you put on your site to tell search robots which pages you would like them not to visit

nilam-shah
11-11-2014, 12:39 AM
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.....

GeethaN
02-20-2021, 01:19 AM
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.