A survey of Web crawlers for information retrieval

Abstract

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218 This article is categorized under: Algorithmic Development > Web Mining Fundamental Concepts of Data and Knowledge > Information Repositories Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing

“…Seminal research into the graph-theoretical characteristics of the Web simplified its representation by assuming static relationships (edges) between static pages (nodes). Increasingly sophisticated Web crawlers [25] were deployed to traverse hyperlinkage structures, providing insights into its geometric characteristics [9]. Subsequent research has emphasized the importance of studying Internet topology [18,31,54].…”

“…The nature, structure, and influence of the Web have been subject to an overwhelming body of research. However, its rapid rate of evolution and sheer magnitude have outpaced even the most advanced tools used in its study [25,44]. The expansive scale of the modern Web has necessitated increasingly sophisticated strategies for its traversal [6,10].…”

“…However, little is known about the representativeness of such crawls compared to the experience of human users browsing the Web, nor about how much they depend on the environment from which they are run. In order to contextualize the information obtained from Web crawls, this study evaluates the efficacy, reliability, and generalisability of the use of Web crawlers [25] as a surrogate for user interaction with the Web in the context of online tracking. The main contributions of this paper are:…”

Web Crawling and Processing with Limited Resources for Business Intelligence and Analytics Applications

“…Despite simple in principle, the variety of purposes for which data is collected as well as technical issues, made crawling a rich and complex research field. As a measure of the interest for this topic, in a recent survey [4] the authors claim they censused 1488 articles about crawling. Restricting to the real implementations the authors listed 62 works.…”

Data Visualization Analysis of Big Data Recruitment Positions in Hangzhou Based on Python

“…In the case of continuously raising of the requirements of big data, however, talent for big data is in short supply, there are many online recruitment website, for instance, 51job.com [2], ChinaHR, Zhaopin.com [3], www.lagou.com, these websites have become the carrier of a large amount of recruitment information. In the ocean of Web, finding information is like finding a needle in the haystack [4]. The search engine is used to find information on the Web.…”