PDA

View Full Version : What are Spiders, Robots and Crawlers and what are their functions?



marisalewis24
08-01-2016, 06:54 PM
friends tell me what are Spiders, Robots and Crawlers and what are their functions.?

infoseair
08-02-2016, 07:02 AM
These three are same. This is search engine robot to crawling and indexing on a web page.

mani ge3e
08-02-2016, 09:01 AM
The search engine spiders, crawl throughout the web. The content they come across are indexed in their massive search indices. The webpages that are indexed, are grouped according to their authority, trust, high-quality content and the number of backlinks.

The job of these search spiders is to crawl the entire web 24 x 7.

Another term to take account of is, crawl rate. Crawl rate refers to the rate at which the search spiders crawl a particular site or blog.
Any authoritative sites like NY Times, Forbes have highly crawl rates. Because they are authoritative and the content updation is too frequent.

However, the question may arises. How do search spiders crawl the Internet?
First they make a list of highly authoritative sites like NY Times, Forbes, TechCrunch, BBC News, etc. they crawl these sites first. And then they move on for crawling the backlinks, in those authoritative sites. Like this then make use of the backlinks from all the sites (starting from authoritative sites) to crawl the entire web.

While crawling they analyze the quality and authority of the blog content. And take that into account while placing the results in SERPS.