Techniques and Applications for Crawling, Ingesting and Analyzing Blockchain Data

Abstract

As the public Ethereum network surpasses half a billion transactions and enterprise Blockchain systems becoming highly capable of meeting the demands of global deployments, production Blockchain applications are fast becoming commonplace across a diverse range of business and scientific verticals. In this paper, we reflect on work we have been conducting recently surrounding the ingestion, retrieval and analysis of Blockchain data. We describe the scaling and semantic challenges when extracting Blockchain data in a way that preserves the original metadata of each transaction by cross referencing the Smart Contract interface with the on-chain data. We then discuss a scientific use case in the area of Scientific workflows by describing how we can harvest data from tasks and dependencies in a generic way. We then discuss how crawled public blockchain data can be analyzed using two unsupervised machine learning algorithms, which are designed to identify outlier accounts or smart contracts in the system. We compare and contrast the two machine learning methods and cross correlate with public Websites to illustrate the effectiveness such approaches.

EOS.IO blockchain data analysis

“…This type of work often first needs to analyse the original data and then cluster, train the model through supervised learning or unsupervised learning, or a combination of the two, and then perform subsequent prediction work. Another example is the literature [59]. After describing the process of gaining and retrieving data, it discussed how to use two unsupervised machine learning algorithms to analyse the gained public chain data, found some outliers and smart contracts, and then compared the two machine learning methods cross-linked with public websites to illustrate the effectiveness of this method.…”

Blockchain Data Analysis from the Perspective of Complex Networks: Overview

“…For instance, in Ref. [68], after the process of obtaining and retrieving data is described, how to use two unsupervised machine learning algorithms to analyze the acquired common chain data was discussed, and some outliers and intelligent contracts were found. Then, they compared the effectiveness of this method by comparing the two machine learning methods and cross-linking them with public websites.…”

Blockchain logging for process mining: a systematic review

“…Similarly the approach in [28] tackles the problem by controlling how the logs are created and stored into the blockchain. Their focus is on science processes, similar to business processes in that they can be modeled as self-contained tasks with specific data dependency and execution logic.…”