Crawling and Analysis of Dark Network Data

Yang, Ying; Zhu, Guichun; Yang, Lina; Yu, Huanhuan

doi:10.1145/3379247.3379272

Cited by 7 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Accuracy testing was employed to evaluate the URL content obtained from crawling in relation to the input keywords. This analysis also considered the impact of Regex Text from research [13], Regex Wildcard, and Regex Option on the collected data. The accuracy analysis was performed on the data obtained from the same scenario as the performance testing, where each URL was tested ten times using three distinct keyword processing methods.…”

Section: Accuracy Testingmentioning

confidence: 99%

“…Most research in this area proposed a crawler that digs down the dark web to obtain a comprehensive database for topic profiling and classification. This kind of research can be seen in research of Pannu et al [12] for creating a database of suspicious websites, Alkhatib et al [6] for market structure and product summary, Yang et al [13] for hidden threat intelligence, Lee at el. [14] to uncover types of cyber-criminal activities occurring in South Korea, Shiaeles et al [15] for monitoring and analysis of attack trends in the IoT ecosystem, Shinde et al [16] to uncover child and women abuse materials, Alharbi et al [17] to analyze the Tor dark web graph's internal structure and connectivity.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Performance Analysis of Regex-Based Processing for Dark Web Targeted Crawling

Ruriawan,

Purwanto,

Yunelfi

et al. 2024

IJSSE

View full text Add to dashboard Cite

Data crawling in the dark web holds a critical significance in bolstering security intelligence efforts. Previous research has successfully developed fast crawlers for specific purposes such as digital investigations, abusive content, automated captcha breaking, etc. However, this research mostly focuses on faster download time and has not paid attention to the importance of assessing crawl accuracy. Due to the fast-changing dark web shape and content, accurate and complete crawled data is a vital part of security intelligence. This research has successfully developed a targeted dark web crawler by combining the focus and in-depth crawling for The Onion Router (TOR) network. Regex Text, Regex Wildcard, and Regex Optional are used to automatically filter the content by a specific keyword. The effectiveness of the crawler was tested in the five real-world dark website environments. From the testing with a depth of 3, the application achieved more than 98% accuracy. The Regex Optional processing performance was faster than the Regex Text and Regex Wildcard by over a second, due to the swift crawling attempt. In terms of accuracy, the Regex Optional achieved 99.14% which is 4.83% higher than Regex Text. The best keyword processing method in targeted crawling is Regex Optional, with an accuracy rate of over 99%.

show abstract

Section: Accuracy Testingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Performance Analysis of Regex-Based Processing for Dark Web Targeted Crawling

Ruriawan,

Purwanto,

Yunelfi

et al. 2024

IJSSE

View full text Add to dashboard Cite

show abstract

“…Program language: Python. [104] Self-developed N/A Used a Selenium based crawler to classify Tor websites. Program language: Python.…”

Section: Articlementioning

confidence: 99%

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Bergman

Popov

2023

IEEE Access

View full text Add to dashboard Cite

Strong encryption algorithms and reliable anonymity routing have made cybercrime investigation more challenging. Hence, one option for law enforcement agencies (LEAs) is to search through unencrypted content on the Internet or anonymous communication networks (ACNs). The capability of automatically harvesting web content from web servers enables LEAs to collect and preserve data prone to serve as potential leads, clues, or evidence in an investigation. Although scientific studies have explored the field of web crawling soon after the inception of the web, few research studies have thoroughly scrutinised web crawling on the "dark web" or via ACNs such as I2P, IPFS, Freenet, and Tor. The current paper presents a systematic literature review (SLR) that examines the prevalence and characteristics of dark web crawlers. From a selection of 58 peer-reviewed articles mentioning crawling and the dark web, 34 remained after excluding irrelevant articles. The literature review showed that most dark web crawlers were programmed in Python, using either Selenium or Scrapy as the web scraping library. The knowledge gathered from the systematic literature review was used to develop a Tor-based web crawling model into an already existing software toolset customised for ACN-based investigations. Finally, the performance of the model was examined through a set of experiments. The results indicate that the developed crawler was successful in scraping web content from both clear and dark web pages, and scraping dark marketplaces on the Tor network. The scientific contribution of this paper entails novel knowledge concerning ACN-based web crawlers. Furthermore, it presents a model for crawling and scraping clear and dark websites for the purpose of digital investigations. The conclusions include practical implications of dark web content retrieval and archival, such as investigation clues and evidence, and the related future research topics.INDEX TERMS cybercrime, digital forensics, systematic literature review, dark web crawling, Tor

show abstract