Systematic Construction, Execution, and Reproduction of Complex Performance Benchmarks

Lima, Rodrigo Alves; Kimball, Joshua; Ferreira, João Eduardo; Pu, Calton

doi:10.1007/978-3-030-23502-4_3

Cited by 3 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Han et al [1] have recently reported their experiences reproducing several performance bugs in server applications, failing to reproduce most of them based on developers' descriptions. In an attempt to help in the replication of complex performance benchmarks, Lima et al [14], developed a framework to collect fine-grained data from executions.…”

Section: Related Workmentioning

confidence: 99%

“…Analogously to the guidelines to document the bugs, templates should be designed to specify the inputs, scripts, configuration options, software and hardware requirements and the step-by-step process to be able to repeat the same environment. Some promising advances have been made towards this direction recently [14]. How to reproduce the most complex bugs or those manifesting under unusual circumstances as well as the development of standard and publicly available repositories containing reproducible performance problems are challenges to be explored in the future.…”

Section: B: Challenge 2: Reproducibility Of Performance Bugsmentioning

confidence: 99%

“…This makes extremely difficult to fully under-stand or reproduce the causes that led to an underperforming system. Reproducibility of performance bugs has recently been identified as an important issue by Lima et al [14] and also by Han et al [1], who, despite their best efforts, were only able to reproduce 17 out of 93 real performance bugs reported and described by developers.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

TANDEM: A Taxonomy and a Dataset of Real-World Performance Bugs

et al. 2020

View full text Add to dashboard Cite

The detection of performance bugs, like those causing an unexpected execution time, has gained much attention in the last years due to their potential impact in safety-critical and resourceconstrained applications. Much effort has been put on trying to understand the nature of performance bugs in different domains as a starting point for the development of effective testing techniques. However, the lack of a widely accepted classification scheme of performance faults and, more importantly, the lack of well-documented and understandable datasets makes it difficult to draw rigorous and verifiable conclusions widely accepted by the community. In this paper, we present TANDEM, a dual contribution related to real-world performance bugs. Firstly, we propose a taxonomy of performance bugs based on a thorough systematic review of the related literature, divided into three main categories: effects, causes and contexts of bugs. Secondly, we provide a complete collection of fully documented real-world performance bugs. Together, these contributions pave the way for the development of stronger and reproducible research results on performance testing. INDEX TERMS performance bugs, performance testing, dataset, taxonomy.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: B: Challenge 2: Reproducibility Of Performance Bugsmentioning

confidence: 99%