Exploiting Routing Information Encoded into Backlinks to Improve Topical Crawling

Mouton, Alban; Marteau, Pierre-François

doi:10.1109/socpar.2009.129

Cited by 3 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, according to the needs of the project, this article only needs to crawl the carpool data of one City and do not considerate a distributed architecture. For non-distributed web crawlers, a method [8] was developed that uses link analysis to determine what constitutes a good content analysis metric. The routing information encoding into backlinks also improved topical crawling.…”

Section: Research Status Of Web Crawlersmentioning

confidence: 99%

Design and Implementation of Carpool Data Acquisition Program Based on Web Crawler

Zhao¹,

Li²

2018

IJMECH

View full text Add to dashboard Cite

Now the public traffics make the life more and more convenient. The amount of vehicles in large or medium sized cities is also in the rapid growth. In order to take full advantage of social resources and protect the environment, regional end-to-end public transport services are established by analyzing online travel data. The usage of computer programs for processing of the web page is necessary for accessing to a large number of the carpool data. In the paper, web crawlers are designed to capture the travel data from several large service sites. In order to maximize the access to traffic data, a breadth-first algorithm is used. The carpool data will be saved in a structured form. Additionally, the paper has provided a convenient method of data collecting to the program.

show abstract

Section: Research Status Of Web Crawlersmentioning

confidence: 99%

Design and Implementation of Carpool Data Acquisition Program Based on Web Crawler

Zhao¹,

Li²

2018

IJMECH

View full text Add to dashboard Cite

show abstract

“…However, once this kind of method provides a clear statement about the assumptions, it also can be a powerful model and have good performance. Mouton and Marteau [26] exploited link analysis to improve topical Web crawling, which is also a probabilistic model. The topical graphs on the Web could guide the crawler toward highly relevant areas of the Web.…”

Section: Related Workmentioning

confidence: 99%

Clustering-Based Topical Web Crawling for Topic-Specific Information Retrieval Guided by Incremental Classifier

Peng

Liu

2015

Int. J. Soft. Eng. Knowl. Eng.

View full text Add to dashboard Cite

Today more and more information on the Web makes it di±cult to get domain-speci¯c information due to the huge amount of data sources and the keywords that have few features. Anchor texts, which contain a few features of a speci¯c topic, play an important role in domain-speci¯c information retrieval, especially in Web page classi¯cation. However, the features contained in anchor texts are not informative enough. This paper presents a novel incremental method for Web page classi¯cation enhanced by link-contexts and clustering. Directly applying the vector of anchor text to a classi¯er might not get a good result because of the limited amount of features. Link-context is used¯rst to obtain the contextual information of the anchor text. Then, a hierarchical clustering method is introduced to cluster feature vectors and content unit, which increases the length of a feature vector belonging to one speci¯c class. Finally, incremental SVM is proposed to get the¯nal classi¯er and increase the accuracy and e±ciency of a classi¯er. Experimental results show that the performance of our proposed method outperforms the conventional topical Web crawler in Harvest rate and Target recall.

show abstract

A survey of Web crawlers for information retrieval

Kumar

Bhatia

Rattan

2017

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218 This article is categorized under: Algorithmic Development > Web Mining Fundamental Concepts of Data and Knowledge > Information Repositories Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

show abstract

Exploiting Routing Information Encoded into Backlinks to Improve Topical Crawling

Cited by 3 publications

References 8 publications

Design and Implementation of Carpool Data Acquisition Program Based on Web Crawler

Design and Implementation of Carpool Data Acquisition Program Based on Web Crawler

Clustering-Based Topical Web Crawling for Topic-Specific Information Retrieval Guided by Incremental Classifier

A survey of Web crawlers for information retrieval

Contact Info

Product

Resources

About