A Novel Web Scraping Approach Using the Additional Information Obtained From Web Pages

Abstract

Web scraping is a process of extracting valuable and interesting text information from web pages. Most of the current studies targeting this task are mostly about automated web data extraction. In the extraction process, these studies first create a DOM tree and then access the necessary data through this tree. The construction process of this tree increases the time cost depending on the data structure of the DOM Tree. In the current web scraping literature, it is observed that time efficiency is ignored. This study proposes a novel approach, namely UzunExt, which extracts content quickly using the string methods and additional information without creating a DOM Tree. The string methods consist of the following consecutive steps: searching for a given pattern, then calculating the number of closing HTML elements for this pattern, and finally extracting content for the pattern. In the crawling process, our approach collects the additional information, including the starting position for enhancing the searching process, the number of inner tag for improving the extraction process, and tag repetition for terminating the extraction process. The string methods of this novel approach are about 60 times faster than extracting with the DOM-based method. Moreover, using these additional information improves extraction time by 2.35 times compared to using only the string methods. Furthermore, this approach can easily be adapted to other DOM-based studies/parsers in this task to enhance their time efficiencies. INDEX TERMS Computational efficiency, algorithm design and analysis, web crawling and scraping, document object model.

Automatically Discovering Relevant Images From Web Pages

“…In our study, a web crawler was developed that can easily create a dataset and extract new features from web pages. A crawler offers a lot of information about web data extraction [27]. With this crawler, a large dataset of 20,000 web pages from 200 websites was created.…”

“…Uzun et al [5] develop an intelligent crawler, namely iCrawler, automatically pulls content out of various layouts for improving the crawling process. Uzun [27] proposes a novel approach that extracts data quickly using the string functions and additional information including the starting position, the number of the inner tag, and tag repetition obtained from web pages. The data obtained through web crawlers can be used for many different purposes.…”

“…Unfortunately, these features are not enough for accurate prediction. In order to resolve this situation, we identified features that can be obtained from the web page by considering other features [5], [10] suggested in the literature. Furthermore, new features are suggested thanks to the additional modules added to our crawler in this study.…”

Taiwan Stock Tape Reading Periodically Using Web Scraping Technology with GUI

“…Singrodia et al [10] introduced the concept of web scraping from easy to hard. Uzun [11] presented the concept of a document object model tree to scrape website data. Pujari et al [12] selected the XAMPP platform as the display surface to implement a web scraping scene application.…”

Intelligent and adaptive web data extraction system using convolutional and long short-term memory deep learning networks

“…Recent advancements in machine learning and Artificial Intelligence (AI) have unfolded new opportunities, even in extensively studied research programs in numerous domains, including medical imaging (e.g., image recognition), transportation (feature extraction in selfdriving cars) [1,2] , and traffic scenarios (e.g., object detection) [3,4] . These advancements also encourage the extraction of relevant information from documents (pdf, doc, or txt files), websites, and images that use Optical Character Recognition (OCR) [5] , subsequently inspiring the development of automated web data extraction systems through leading edge technology solutions [6,7] . The application of deep learning in web data extraction [8,9] is still in its nascent stage; in addition to extracting data from documents or web pages, this application involves navigating different websites and storing data for analytics and visualization purposes.…”