Review of Deep Web Data Extraction

Li, Shenglin; Chen, Chen; Luo, Kaiwen; Song, Bo

doi:10.1109/ssci44817.2019.9002877

Cited by 8 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has many synonyms such as web scraping, web crawling, data extraction, information extraction etc. For consistency in this study the authors use web data extraction including web page navigation, data extraction, enrichment and visualization [31]. With rapid developments in web applications technologies extracting data from web became increasingly challenging.…”

Section: Data Extraction: Background and Evolutionmentioning

confidence: 99%

A Web Information Extraction Framework with Adaptive and Failure Prediction Feature

Patnaik

Babu

2022

J. Data and Information Quality

View full text Add to dashboard Cite

The amount of information available on the internet today requires effective information extraction and processing to offer hyper personalized user experiences. Inability to extract information by using traditional and machine learning techniques due to dynamic changes in website layout pose significant challenge to the technical community to keep up to such changes. The focus of existing machine learning based information extraction framework is only on information extraction by using core extraction logic that is susceptible to website changes, thus missing out core features such as ability to handle proactive failure prediction and intelligent information extraction capabilities. The aim of the paper is to build a robust and an intelligent information extraction framework with ability not only to proactively predict website failure but also automatically extract information using deep learning techniques using You only look once (Yolo) and Long short-term memory (LSTM) networks. The proactive detection using LSTM detects new location of the web page due to layout changes and enables automatic extraction of information of the new web page. A real-world case with retail website for intelligent information extraction and an offline experimentation environment is setup to demonstrate proactive failure prediction and automatic extraction resulting in high failure prediction, precision and recall of object detection and information extraction.

show abstract

Section: Data Extraction: Background and Evolutionmentioning

confidence: 99%

A Web Information Extraction Framework with Adaptive and Failure Prediction Feature

Patnaik

Babu

2022

J. Data and Information Quality

View full text Add to dashboard Cite

show abstract

“…The process of extraction of data from the dark web is similar to that on the surface web as they both are built using a hypertext markup language (HTML). Several existing works have surveyed the tools and techniques for extracting data from the surface web and deep web [36]- [38]. The data extraction from the web can be performed either manually or using fully automatic methods.…”

Section: Related Workmentioning

confidence: 99%

Exploring the Topological Properties of the Tor Dark Web

et al. 2021

View full text Add to dashboard Cite

The web graph can be used to get insight into the internal structure and connectivity of the Tor dark web. This paper analyzes the internal structure of the Tor dark web graph and examines the presence of bow-tie structure as found in the World Wide Web. The web graph is generated from the data collected by the Python crawler customized to scrape data from the Tor dark web. Each of the nodes in the graph represents an individual Tor hidden service, and an edge denotes the hyperlink from one hidden service to the other. Various graph metrics are then computed and analyzed for both directed and undirected graphs using the Python NetworkX package. It was found that most of the nodes of the graph have in-degree and out-degree less than ten. The presence of power-law in degree distribution could neither be confirmed nor denied. The Tor web graph is sparse with a few connected pairs of nodes. Like the surface web, the dark web can also be decomposed into a bow-tie structure though with small component sizes. Several important and well-known websites on the surface web have incoming links from the dark web. Moreover, the Tor network also shows the characteristics of small-world and scale-free networks.INDEX TERMS Bow tie, dark web, graph analysis, Tor, web graph.

show abstract

“…These advancements also encourage the extraction of relevant information from documents (pdf, doc, or txt files), websites, and images that use Optical Character Recognition (OCR) [5] , subsequently inspiring the development of automated web data extraction systems through leading edge technology solutions [6,7] . The application of deep learning in web data extraction [8,9] is still in its nascent stage; in addition to extracting data from documents or web pages, this application involves navigating different websites and storing data for analytics and visualization purposes.…”

Section: Introductionmentioning

confidence: 99%

Intelligent and adaptive web data extraction system using convolutional and long short-term memory deep learning networks

Patnaik

Babu

Bhave³

2021

Big Data Min. Anal.

View full text Add to dashboard Cite

Data are crucial to the growth of e-commerce in today's world of highly demanding hyper-personalized consumer experiences, which are collected using advanced web scraping technologies. However, core data extraction engines fail because they cannot adapt to the dynamic changes in website content. This study investigates an intelligent and adaptive web data extraction system with convolutional and Long Short-Term Memory (LSTM) networks to enable automated web page detection using the You only look once (Yolo) algorithm and Tesseract LSTM to extract product details, which are detected as images from web pages. This state-of-the-art system does not need a core data extraction engine, and thus can adapt to dynamic changes in website layout. Experiments conducted on real-world retail cases demonstrate an image detection (precision) and character extraction accuracy (precision) of 97% and 99%, respectively. In addition, a mean average precision of 74%, with an input dataset of 45 objects or images, is obtained.

show abstract

Review of Deep Web Data Extraction

Cited by 8 publications

References 19 publications

A Web Information Extraction Framework with Adaptive and Failure Prediction Feature

A Web Information Extraction Framework with Adaptive and Failure Prediction Feature

Exploring the Topological Properties of the Tor Dark Web

Intelligent and adaptive web data extraction system using convolutional and long short-term memory deep learning networks

Contact Info

Product

Resources

About