Efficient Topical Focused Crawling Through Neighborhood Feature

Suebchua, Tanaphol; Manaskasemsak, Bundit; Rungsawang, Arnon; Yamana, Hayato

doi:10.1007/s00354-017-0029-8

Cited by 7 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where f rs is the function for related score of the web page, w p , w blc , w lm , w flc , w au , w ak are the weights associated with the page rank, backward link count, location metric, forward link count, association weight of the URL and keyword, respectively, f p is the page rank, f blc is the backward link count, f lm is the location metric, f flc is the forward link count, f au is the association of the URL with the keyword, and f ak is the association of the keyword with the body. Suebchua et al (2018) discussed a focused crawler using the neighbourhood feature. Linkage features and the neighbourhood features are extracted from the web pages and given as input to three different NB classifiers for prediction.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A supervised learning‐based approach for focused web crawling for IoMT using global co‐occurrence matrix

Rajiv

Navaneethan

2022

Expert Systems

View full text Add to dashboard Cite

Irrelevant search results for a given topic end up wasting search engine users' time. A learning focused web crawler downloads relevant URLs for a given topic using machine‐learning algorithms. The dynamic nature of the web is a challenge in related computation for focused web crawlers. Studies have shown that the learning focused crawler utilizes term frequency‐inverse document frequency (TF‐IDF) to compute the relevance between a web page and a given topic. The TF‐IDF detects similarity of the given topic to its co‐occurrence on the web page. The necessity of efficient mechanism to compute the relevance of URLs syntactically and semantically has led to the proposal of this paper with a word embedding approach to compute the relevance of the web page. The global vector representation cosine similarity is calculated between a topic and the web page contents. The calculated cosine similarity is provided as input to the trained random forest classifier to predict the relevancy of the web page. The evaluation results proved that the proposed crawler produced an average hrate of 0.41 and prate0.25em of 0.59, which outperformed other learning‐focused crawlers on support vector machines, Naive Bayes and artificial neural networks.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Suebchua et al (2018) discussed a focused crawler using the neighbourhood feature. Linkage features and the neighbourhood features are extracted from the web pages and given as input to three different NB classifiers for prediction.…”

Section: Related Workmentioning

confidence: 99%

A supervised learning‐based approach for focused web crawling for IoMT using global co‐occurrence matrix

Rajiv

Navaneethan

2022

Expert Systems

View full text Add to dashboard Cite

show abstract

“…The critic classifies web pages, while the apprentice learns (from the critic's feedback) to distinguish the most promising outlink URLs. Suebchua et al [10] introduced the "Neighborhood feature", which exploits the relevance of all already fetched web pages of the same domain (web site), in order to more effectively select URLs belonging to the same domain. This approach assumes that a web page is likely to be relevant if many relevant web pages have been discovered in its neighborhood, which is similar to the empirical evidence of topical locality on the Web showed by Davison [11].…”

Section: Related Workmentioning

confidence: 99%

“…Unlike common (but more unrealistic) approaches [10,17], for each experiment we utilize a single seed and average results from 10 different single-seed crawling runs. To select seeds, we used URLs from Google, such that they are not immediately mutually connected on the Web.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Tree-based Focused Web Crawling with Reinforcement Learning

Kontogiannis¹,

Kelesis²,

Pollatos³

et al. 2021

Preprint

View full text Add to dashboard Cite

A focused crawler aims at discovering as many web pages relevant to a target topic as possible, while avoiding irrelevant ones; i.e. maximizing the harvest rate. Reinforcement Learning (RL) has been utilized to optimize the crawling process, yet it deals with huge state and action spaces, which can constitute a serious challenge. In this paper, we propose TRES, an end-to-end RL-empowered framework for focused crawling. Unlike other approaches, we properly model a crawling environment as a Markov Decision Process, by representing the state as a subgraph of the Web and actions as its expansion edges. TRES adopts a keyword expansion strategy based on the cosine similarity of keyword embeddings. To learn a reward function, we propose a deep neural network, called Kw-BiLSTM, leveraging the discovered keywords. To reduce the time complexity of selecting a best action, we propose Tree-Frontier, a two-fold decision tree, which also speeds up training by discretizing the state and action spaces. Experimentally, we show that TRES outperforms state-of-the-art methods in terms of harvest rate by at least 58%, while it has competitive results in the domain maximization. Our implementation code can be found on https://github.com/ddaedalus/TRES.

show abstract