First-order focused crawling

Xu, Qingyang; Zuo, Wangmeng

doi:10.1145/1242572.1242744

Cited by 13 publications

(6 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These techniques do not use strict mathematical definitions and are tolerant to imprecision, uncertainty, and partial truth to achieve a solution. In the starting paragraph of each category, we will discuss common characteristics of that technique and their subsequent 109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,<...>…”

Section: Focused Web Crawler Using Soft Computing Techniquesmentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218 This article is categorized under: Algorithmic Development > Web Mining Fundamental Concepts of Data and Knowledge > Information Repositories Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

show abstract

Section: Focused Web Crawler Using Soft Computing Techniquesmentioning

confidence: 99%

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

show abstract

“…Ari Pirkola in [32] described negligence of historical results and inability to handle intermediate linguity as the main problems for any crawler. Xu, Qingyang and Zuo, Wanli [33] presented general framework of focused web crawling based on "relational subgroup discovery". Predicates were used explicitly to represent the relevance clues of those unvisited pages in the crawl frontier, and then first-order classification rules were induced using subgroup discovery technique.…”

Section: Related Workmentioning

confidence: 99%

Focused Crawling Based Upon TF-IDF Semantics and Hub Score Learning

Kumar¹,

Vig²

2013

JETWI

View full text Add to dashboard Cite

A focused crawler traverses the Web to collect documents related to a particular topic, and can be used to build topic specific collection of documents for use in digital libraries and domain specific search. General crawlers make use of breath first search method to traverse the Web for as much amount of information as possible. Focused crawler help the search indexer to index all documents present on the World Wide Web related to a specific domain which in turn provides search engine’s users complete and fresher most information. In this paper we present a focused crawler capable of learning from the previous crawl results to collect the documents related to the sports domain. Crawling results for four consecutive crawls are shown. Results shows significant improvement in the precision value for the crawler with respect to the number of crawling attempts made

show abstract

“…Some efforts by different researchers in the same direction are given below. Focused crawler [11] discover semantic web data, by using some sort of heuristic to rate pages according to their relevance to a given topic. This crawler should stay focused around the given topic, so that irrelevant pages should not pursued by the crawler.…”

Section: Related Workmentioning

confidence: 99%

A Novel Architecture of Ontology-based Semantic Web Crawler

Rana¹,

Tyagi²

2012

IJCA

View full text Add to dashboard Cite

Finding meaningful information among the billions of information resources on the Web is a difficult task due to growing popularity of the Internet. The future of World Wide Web (WWW) is the Semantic Web, where ontologies are used to assign (agreed) meaning to the content of the Web. On the Semantic Web, data will inevitably be linked to many different ontologies, and information processing across ontologies is not possible without knowing the semantic mappings between them. As the resources on the Semantic Web are annotated using these ontologies, new search techniques are required to find specific information. For this, architecture has been proposed for ontology based semantic web crawler. This architecture can exploit the semantic metadata to efficiently discover and extract information from the Semantic Web. In this paper Semantic matching between content of downloaded web page and ontology is used to guide the crawler towards relevant information.

show abstract

First-order focused crawling

Cited by 13 publications

References 4 publications

A survey of Web crawlers for information retrieval

A survey of Web crawlers for information retrieval

Focused Crawling Based Upon TF-IDF Semantics and Hub Score Learning

A Novel Architecture of Ontology-based Semantic Web Crawler

Contact Info

Product

Resources

About