Results 1 to 11 of 11
  1. #1

  2. #2
    Registered User
    Join Date
    Dec 2018
    Posts
    643
    A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."

  3. #3
    Senior Member
    Join Date
    Jun 2019
    Posts
    206
    A web crawler is a computer program that is used to search through millions of websites that are available on the internet according to a search query provided by the user. They may use any of the algorithms that can return the list of websites that are relevant with the query like A star, Page rank etc.

  4. #4
    Senior Member
    Join Date
    Sep 2019
    Posts
    771
    A crawler is a computer program that automatically searches documents on the Web. Crawlers are primarily programmed for repetitive actions so that browsing is automated. Search engines use crawlers most frequently to browse the internet and build an index.

  5. #5
    Registered User
    Join Date
    Aug 2018
    Location
    Scottsdale, Arizona
    Posts
    72
    The job of a web crawler is to read the activity on the website and tell the search engine. Like Google's Google crawler which we also know by the name of robot.txt file. It crawls the pages of your website and indexes your website. Your website will not be indexed in search engines until this crawler crawls your website.

  6. #6
    Registered User
    Join Date
    May 2015
    Posts
    524
    A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."

  7. #7
    Senior Member
    Join Date
    Nov 2018
    Posts
    1,348
    A web crawler is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or spidering. Many legitimate sites, in particular, search engines, use spidering as a means of providing up-to-date data.

  8. #8

  9. #9
    Registered User
    Join Date
    Oct 2019
    Location
    United Arab Emirates
    Posts
    31
    A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.

  10. #10
    Registered User
    Join Date
    Dec 2018
    Posts
    643
    A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. ... Crawlers apparently gained the name because they crawl through a site a page at a time, following the links to other pages on the site until all pages have been read.

  11. #11
    Senior Member
    Join Date
    Jul 2018
    Posts
    454
    A web crawler is a computer program that is used to search through millions of websites that are available on the internet according to a search query provided by the user. They may use any of the algorithms that can return the list of websites that are relevant with the query like A star, Page rank etc.

    The term crawling literally means scanning through any piece of text, be it links to other websites, image captions, descriptions, articles, anything. It's like a spider searching through the web.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.