What do you mean by a crawler?

[tony_S]

What do you mean by a crawler?

[bessieexum]

There is a web crawler an online bot which helps in Web indexing. One page crawls at one time until all pages have been indexed. Web-crawlers links and assist in collecting information regarding also the links related to them, and help in supporting the code and a website.

A Web crawler is also called a Web spider indexer or simply just crawler.

[Neo_5678]

A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."

[nancy07]

The crawler is bots by the search engine which helps search engine to index website in their database and also to rank according to website keyword optimization.
Crawler read the website and let the search engine know about the website category.

[Ravina97]

A crawler is a program that visits websites and reads their pages and other information in an effort to create entries for a search engine index. The serps on the internet all have this kind of program, which is also known as a "spider" or a "bot." Crawlers are more often than not programmed to talk over with websites that have been submitted by using their owners as new or updated. Entire web sites or specific pages will also be selectively visited and indexed.A crawler is also known as a Web spider, automatic indexer or bot.

[Wiztechplc]

Crawlers are bots that run automatically all over the web to index new content that is available on the web.

Example:

Google Bot
MSNBot
Yahoo Slurp

[yuva12]

A Web Crawler is a program which is used by search engine to find what is new on the Internet. This process is called Crawling.

[archanaluthra]

A crawler is a program that accesses a website and reads its pages and other information to create an entry for a search engine index. The main search engines on the Internet have such a program, also known as "spider" or "robot."

[stellasweety]

There is a web crawler an online bot which helps in Web indexing. One page crawls at one time until all pages have been indexed. Web-crawlers links and assist in collecting information regarding also the links related to them, and help in supporting the code and a website.

[Ravina97]

A crawler is a software used by serps to acquire information from the internet. When a crawler visits a website, it picks over the entire website’s content material (i.e. The textual content) and stores it in a database. It additionally stores all the external and internal links to the site. The crawler will visit the stored links at a later point in time, which is how it moves from one site to the next. By means of this process the crawler captures and indexes every website that has links to at the least one different website.

[Sara James]

Stop making the post count

[beautyepic]

In search engine terminology crawler is popular by various names like robot, bot, spider etc. Spider is standard program consisting of many algorithms which scans almost all Web pages on world wide websites. How spider works is very difficult and complicated thing to understand but it is said that spider hops from one Web page to other according to links provided on Web page.

[dombowkett]

Each search engine have their own web crawler to index the website.

[SKS]

It is a web program that is designed for performing the specific task to visit newly made web pages and the updated web pages to read the content and put the web page link in the SE database for indexing. Crawler also known as web crawler, bot or robot.

[Jessicad0505]

Crawlers collect information such the URL of the website, the meta tag information, the Web page content, the links in the webpage and the destinations leading from those links, the web page title and any other relevant information.