Gromit: Benchmarking the Performance and Scalability of Blockchain Systems

Nasrulin, Bulat; Vos, Martijn de; Ishmaev, Georgy; Pouwelse, Johan

doi:10.1109/dapps55202.2022.00015

Cited by 7 publications

(5 citation statements)

References 23 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…also undermine the role of the clients as the ultimate recipient. This clearly shows the difference to other measurements obtained from papers comparing multiple blockchain systems[25,39,45].Like Corda Enterprise, Quorum shows a downward trend in MTPS values from eight nodes onwards due to the communication overhead of the nodes and the consensus algorithm.…”

contrasting

confidence: 78%

“…In most comprehensive blockchain benchmarking papers ( [45], [25,38], [51]), the performance evaluation is based on the block generation time in the blockchain. Whether the blockchain application receives confirmation of transaction execution is not further investigated.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…Nasrulin et al [45] present Gromit, a performance evaluation framework which they use to analyse seven different systems. Like us, they analyse Fabric version 1.4.9, Diem and BitShares, among others.…”

Section: Related Workmentioning

confidence: 99%

“…Our research uniquely positions itself within the broader landscape of related works. While we introduce a custom framework and evaluate multiple blockchain systems as seen in references like [25,39,45], our study stands out due to its concentrated focus on end-to-end scenarios. Though we recognize and appreciate the contributions of prior studies, our analysis suggests that most have not centrally emphasized this aspect.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

An End-to-End Performance Comparison of Seven Permissioned Blockchain Systems

Geyer,

Jacobsen,

Mayer

et al. 2023

Proceedings of the 24th International Middleware Conference on ZZZ

View full text Add to dashboard Cite

The emergence of numerous blockchain solutions, offering innovative approaches to optimise performance, scalability, privacy, and governance, complicates performance analysis. Reasons for the difficulty of benchmarking blockchains include, for example, the high number of system parameters to configure and the effort to deploy a blockchain network. In addition, performance data, which mostly comes from system vendors, is often opaque. We provide a reproducible evaluation of the performance of seven permissioned blockchain systems across different parameter settings. We employ an end-to-end approach, where the clients sending the transactions are fully involved in the data collection approach. Our results underscore the unique characteristics and limitations of the systems we examined. Due to the insights given, our work forms the basis for continued research to optimise the performance of blockchain systems. CCS CONCEPTS• Computer systems organization → Peer-to-peer architectures.

show abstract

contrasting

confidence: 78%

Section: Evaluation Metricsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

An End-to-End Performance Comparison of Seven Permissioned Blockchain Systems

Geyer,

Jacobsen,

Mayer

et al. 2023

Proceedings of the 24th International Middleware Conference on ZZZ

View full text Add to dashboard Cite

show abstract

“…In contrast to that, the deployment of novel system designs is a common practice in the blockchain industry[18]. However, the quality of experimental evaluations is lacking compared to academic research, as industry whitepapers often present biased and inflated results[20] 3. See https://github.com/ethereum/ropsten/blob/master/revival.md.PerFail 2023: Second International Workshop on Negative Results in Pervasive Computing…”

mentioning

confidence: 99%

A Deployment-First Methodology to Mechanism Design and Refinement in Distributed Systems

Vos

Ishmaev

Pouwelse

et al. 2023

2023 IEEE International Conference on Pervasive Computing and Communications Workshops and Other Affiliated Events (PerCom Work

Self Cite

View full text Add to dashboard Cite

Catalyzed by the popularity of blockchain technology, there has recently been a renewed interest in the design, implementation and evaluation of decentralized systems. Most of these systems are intended to be deployed at scale and in heterogeneous environments with real users and unpredictable workloads. Nevertheless, most research in this field evaluates such systems in controlled environments that poorly reflect the complex conditions of real-world environments. In this work, we argue that deployment is crucial to understanding decentralized mechanisms in a real-world environment and an enabler to building more robust and sustainable systems. We highlight the merits of deployment by comparing this approach with other experimental setups and show how our lab applied a deployment-first methodology. We then outline how we use Tribler, our peer-to-peer file-sharing application, to deploy and monitor decentralized mechanisms at scale. We illustrate the application of our methodology by describing a deployment trial in experimental tokenomics. Finally, we summarize four lessons learned from multiple deployment trials where we applied our methodology.

show abstract