Results 1 to 6 of 6
  1. #1
    Junior Member
    Join Date
    Mar 2010
    Posts
    1

    What Are Search Engine Spiders?

    Search engine spiders are software programs that sift through content on web pages and build lists of words that appear on those pages. This process is called web crawling. The program visits page after page, following each link and recording the content of each page as it goes along like a spider crawling through the pages. This content is then added to the search engine's index.

    Different search engines use different approaches when sending out their search engine spiders. Some keep track of every word on a page. Others record meta tags, titles and subtitle words. Indexing the 100 most common words on a page is another tactic used by a search engine spider.

    Links to websites are what feed search engine spiders. The more often that spiders see links to a website, the more often they visit. This gives the spiders more information to index a site and makes the site appear higher in the search engine's results for a search terms related to that site.

  2. #2
    Senior Member
    Join Date
    Feb 2007
    Posts
    294
    In the past the search spiders were functioning irrelevant of the links they would find across the net but nowadays they are following the backlinks established over the net to find some new websites launched or index recently set up pages.

  3. #3
    Registered User
    Join Date
    Feb 2010
    Posts
    50
    Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code.

  4. #4
    Junior Member
    Join Date
    Apr 2010
    Posts
    17
    A spider, also known as a robot or a crawler, is actually just a program that follows, or "crawls", links throughout the Internet, grabbing content from sites and adding it to search engine indexes.

    Spiders only can follow links from one page to another and from one site to another. That is the primary reason why links to your site are so important. Links to your website from other websites will give the search engine spiders more "food" to chew on. The more times they find links to your site, the more times they will stop by and visit. Google especially relies on its spiders to create their vast index of listings.

    Spiders find Web pages by following links from other Web pages, but you can also submit your Web pages directly to a search engine or directory and request a visit by their spider. In fact, it's a good idea to manually submit your site to a human-edited directory such as Yahoo, and usually spiders from other search engines will find it and add it to their database. It can be useful to submit your URL straight to the various search engines as well; but spider-based engines will usually pick up your site regardless of whether or not you've submitted it to a search engine.

  5. #5
    Junior Member
    Join Date
    Apr 2010
    Posts
    2
    Sometimes they're call spiders, other times they're called bots, and still sometimes they're called crawlers. Either way, they all mean the same thing - programs that search through your website to index it properly.

  6. #6
    Guest Moderator ~ServerPoint~'s Avatar
    Join Date
    Nov 2007
    Posts
    1,846
    Thanks for your information. The thread is closed.
    ServerPoint.com - a true hosting company offering online presence solutions since 1998
    Web Hosting, colocation, dedicated servers, Virtual Private Server (VPS) hosting
    Wholly owned multi homed network, servers and facilities

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

  Find Web Hosting      
  Shared Web Hosting UNIX & Linux Web Hosting Windows Web Hosting Adult Web Hosting
  ASP ASP.NET Web Hosting Reseller Web Hosting VPS Web Hosting Managed Web Hosting
  Cloud Web Hosting Dedicated Server E-commerce Web Hosting Cheap Web Hosting


Premium Partners:


Visit forums.thewebhostbiz.com: to discuss the web hosting business, buy and sell websites and domain names, and discuss current web hosting tools and software.