An Enhanced Focused Web Crawler for Biomedical Topics Using Attention Enhanced Siamese Long Short Term Memory Networks

Abstract

The Internet is chosen to be one among the primary source of biomedical information. To retrieve necessary biomedical information, the search engine needs an efficient, focused crawler mechanism. But the area of research concerned with the focused crawler for biomedical topics is notably scanty. However, the quantity, momentum, diversity, and quality of the available online biomedical information, challenges and calls for enhanced aid to crawl. This paper surmounts the challenges and proposes a new learning approach for focused web crawling adopting Attention Enhanced Siamese Long Short Term Memory (AE-SLSTM) Networks with peephole connections which predicts topical relevance of the web page. The proposed AE-SLSTM model accurately computes the semantic similarity between the topic and the web pages. The performance of the newly designed crawler is assessed using two well known metrics namely harvest rate (ℎ 𝑟𝑎𝑡𝑒 ) and irrelevance ratio (𝑝 𝑟𝑎𝑡𝑒 ). The presented crawler surpass the existing focused crawlers with an average ℎ 𝑟𝑎𝑡𝑒 of 0.39 and an average 𝑝 𝑟𝑎𝑡𝑒 of 0.61 after crawling 5,000 web pages relating to biomedical topics. The results clearly depicts that the proposed methodology aids to download more relevant biomedical web pages related to the particular topic from the internet. HIGHLIGHTS This paper proposes a new focused crawler for biomedical topics.  This paper proposes a novel Attention Enhanced Siamese Long Short Term Memory Networks.  The proposed model is trained using ADAM optimizer with Batch Normalization.  This paper produces an average harvest rate of 0.39.2

An Automated Customizable Live Web Crawler for Curation of Comparative Pharmacokinetic Data: An Intelligent Compilation of Research-Based Comprehensive Article Repository

“…A biomedical web crawler currently in use can predict the importance of a website based on the topic of the website it addresses [ 82 ]. Other web crawlers from the PubMed search follow by URL identification and article download using the GNU Wget web crawler [ 83 , 84 ], as well as online visibility assessment of Biomedical Text Mining (BioTM) tools and resources with JSOUP java API [ 85 , 86 ].…”