Python Scrapers for Scraping Cryptomarkets on Tor

Wu, Yubao; Zhao, Fengpan; Chen, Xucan; Skums, Pavel; Sevigny, Eric L.; Maimon, David; Ouellet, Marie; Swahn, Monica H.; Strasser, Sheryl; Feizollahi, Mohammad Javad; Zhang, Youfang; Sekhon, Gunjan

doi:10.1007/978-3-030-24907-6_19

Cited by 4 publications

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our team developed unique scrapers in Python for 7 cryptomarkets shown in Table I. After scraping, the data is stored in a MySQL database [29]. The scraper primarily targets publicly accessible web pages, such as product rating pages.…”

Section: Datamentioning

confidence: 99%

Into the Reverie: Exploration of the Dream Market

Carr

Zhuang

Sablan

et al. 2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Since the emergence of the Silk Road market in the early 2010s, dark web 'cryptomarkets' have proliferated and offered people an online platform to buy and sell illicit drugs, relying on cryptocurrencies such as Bitcoin for anonymous transactions. However, recent studies have highlighted the potential for de-anonymization of bitcoin transactions, bringing into question the level of anonymity afforded by cryptomarkets. We examine a set of over 100,000 product reviews from several cryptomarkets collected in 2018 and 2019 and conduct a comprehensive analysis of the markets, including an examination of the distribution of drug sales and revenue among vendors, and a comparison of incidences of opioid sales to overdose deaths in a US city. We explore the potential for de-anonymization of vendors by implementing a Naïve-Bayes classifier to predict the vendor from a given product review, and attempt to link vendors' sales to specific Bitcoin transactions. On the buyer side, we evaluate the efficacy of hierarchical agglomerative clustering for grouping together transactions corresponding to the same buyer. We find that the high degree of specialization among the small subset of high-revenue vendors may render these vendors susceptible to de-anonymization. Further research is necessary to confirm these findings, which are restricted by the scarcity of ground-truth data for validation.

show abstract

Section: Datamentioning

confidence: 99%

Into the Reverie: Exploration of the Dream Market

Carr

Zhuang

Sablan

et al. 2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

Evolution of Dark Web Threat Analysis and Detection: A Systematic Approach

et al. 2020

View full text Add to dashboard Cite

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Bergman

Popov

2023

IEEE Access

View full text Add to dashboard Cite

Strong encryption algorithms and reliable anonymity routing have made cybercrime investigation more challenging. Hence, one option for law enforcement agencies (LEAs) is to search through unencrypted content on the Internet or anonymous communication networks (ACNs). The capability of automatically harvesting web content from web servers enables LEAs to collect and preserve data prone to serve as potential leads, clues, or evidence in an investigation. Although scientific studies have explored the field of web crawling soon after the inception of the web, few research studies have thoroughly scrutinised web crawling on the "dark web" or via ACNs such as I2P, IPFS, Freenet, and Tor. The current paper presents a systematic literature review (SLR) that examines the prevalence and characteristics of dark web crawlers. From a selection of 58 peer-reviewed articles mentioning crawling and the dark web, 34 remained after excluding irrelevant articles. The literature review showed that most dark web crawlers were programmed in Python, using either Selenium or Scrapy as the web scraping library. The knowledge gathered from the systematic literature review was used to develop a Tor-based web crawling model into an already existing software toolset customised for ACN-based investigations. Finally, the performance of the model was examined through a set of experiments. The results indicate that the developed crawler was successful in scraping web content from both clear and dark web pages, and scraping dark marketplaces on the Tor network. The scientific contribution of this paper entails novel knowledge concerning ACN-based web crawlers. Furthermore, it presents a model for crawling and scraping clear and dark websites for the purpose of digital investigations. The conclusions include practical implications of dark web content retrieval and archival, such as investigation clues and evidence, and the related future research topics.INDEX TERMS cybercrime, digital forensics, systematic literature review, dark web crawling, Tor

show abstract

Python Scrapers for Scraping Cryptomarkets on Tor

Cited by 4 publications

References 5 publications

Into the Reverie: Exploration of the Dream Market

Into the Reverie: Exploration of the Dream Market

Evolution of Dark Web Threat Analysis and Detection: A Systematic Approach

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Contact Info

Product

Resources

About