IHWC: intelligent hidden web crawler for harvesting data in urban domains

Abstract

Due to the massive size of the hidden web, searching, retrieving and mining rich and high-quality data can be a daunting task. Moreover, with the presence of forms, data cannot be accessed easily. Forms are dynamic, heterogeneous and spread over trillions of web pages. Significant efforts have addressed the problem of tapping into the hidden web to integrate and mine rich data. Effective techniques, as well as application in special cases, are required to be explored to achieve an effective harvest rate. One such special area is atmospheric science, where hidden web crawling is least implemented, and crawler is required to crawl through the huge web to narrow down the search to specific data. In this study, an intelligent hidden web crawler for harvesting data in urban domains (IHWC) is implemented to address the relative problems such as classification of domains, prevention of exhaustive searching, and prioritizing the URLs. The crawler also performs well in curating pollution-related data. The crawler targets the relevant web pages and discards the irrelevant by implementing rejection rules. To achieve more accurate results for a focused crawl, ICHW crawls the websites on priority for a given topic. The crawler has fulfilled the dual objective of developing an effective hidden web crawler that can focus on diverse domains and to check its integration in searching pollution data in smart cities. One of the objectives of smart cities is to reduce pollution. Resultant crawled data can be used for finding the reason for pollution. The crawler can help the user to search the level of pollution in a specific area. The harvest rate of the crawler is compared with pioneer existing work. With an increase in the size of a dataset, the presented crawler can add significant value to emission accuracy. Our results are demonstrating the accuracy and harvest rate of the proposed framework, and it efficiently collect hidden web interfaces from large-scale sites and achieve higher rates than other crawlers.

Efficient intelligent crawler for hamming distance based on prioritization of web documents

“…In comparison to the usual design, the implemented technique crawls, allowing for more equitable data distribution through plug-and-plug. When compared with the existing methods such as IHCM [16], SVM [17], NTPDCF [18], and SIMHAR [21], the implemented hamming distance method achieves better performance values of 99.8%, 99.9%, 98%, and 99% in terms of accuracy, precision, recall, and f-measure.…”

“…This section provides a discussion of the implemented hamming distance and compares those results with existing methods such as IHCM [16], SVM [17], NTPDCF [18], and SIMHAR [21] in the comparative analysis section. The main goal of the hamming distance is to avoid becoming the bottleneck in the pipeline, these algorithms must be fast and efficient.…”

“…The use of web crawler technology may lead to the development of novel environment survey techniques and approaches while obviating the need for significant amounts of labor, money, and time [15]. Kaur et al [16] implemented an intelligent hidden web crawler (IHCW) for harvesting data in urban domains to handle the relevant issues, including prioritizing URLs, domain classification, and avoiding exhaustive searching. The crawler operates well while collecting data on pollutants.…”

Text Matching Technique-based Intelligent Web Crawler in Hybrid Mode

“…Kaur [14] implemented an intelligent hidden web crawler (IHWC) method to solve the relevant issues such as domain classification, prioritizing URLs, and avoiding exhaustive searching. Using rejection rules, the crawler selects appropriate web pages and ignores the insignificant ones.…”

Effect Analysis of Carbon Information on Enterprise Value Based on Big Data

“…erefore, the direction of the theme crawler is based on the simple crawler, crawling more targeted content, crawling more accurate information in a particular field, for example, using a web crawler to crawl agricultural and financial information, etc. [9][10][11]. In a specific program operation, the topic web crawler will receive program instructions related to the topic before crawling the data.…”