Performance Metamorphic Testing: Motivation and Challenges

Segura, Sergio; Troya, Javier; Durán, Amador; Ruiz–Cortés, Antonio

doi:10.1109/icse-nier.2017.16

Cited by 26 publications

(28 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can be used to compare different programs for the same input and determine whether some relations are kept from one execution to another [35]. Recently, some works discuss the use of metamorphic testing [17], [36] to alleviate the test oracle problem in the context of non functional properties. Applied to our domain, the idea is to specify from a large set of test cases, in our case being input programs, the relation between these inputs and the expected qualitative properties outputs.…”

Section: Software Testingmentioning

confidence: 99%

Empirical Assessment of Multimorphic Testing

Temple

Acher

Jézéquel

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

The performance of software systems such as speed, memory usage, correct identification rate, tends to be an evermore important concern, often nowadays on par with functional correctness for critical systems. Systematically testing these performance concerns is however extremely difficult, in particular because there exists no theory underpinning the evaluation of a performance test suite, i.e., to tell the software developer whether such a test suite is "good enough" or even whether a test suite is better than another one. This paper proposes to apply Multimorphic testing and empirically assess the effectiveness of performance test suites of software systems coming from various domains. By analogy with mutation testing, our core idea is to leverage the typical configurability of these systems, and to check whether it makes any difference in the outcome of the tests: i.e., are some tests able to "kill" underperforming system configurations? More precisely, we propose a framework for defining and evaluating the coverage of a test suite with respect to a quantitative property of interest. Such properties can be the execution time, the memory usage or the success rate in tasks performed by a software system. This framework can be used to assess whether a new test case is worth adding to a test suite or to select an optimal test suite with respect to a property of interest. We evaluate several aspects of our proposal through 3 empirical studies carried out in different fields: object tracking in videos, object recognition in images, and code generators.

show abstract

Section: Software Testingmentioning

confidence: 99%

Empirical Assessment of Multimorphic Testing

Temple

Acher

Jézéquel

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…For performance assessment, MT was also used by Segura et al to reveal performance failures. MRs are used as fitness functions to guide the search‐based algorithms in the context of the automated analysis of feature models.…”

Section: Related Workmentioning

confidence: 99%

Leveraging metamorphic testing to automatically detect inconsistencies in code generator families

Boussaa

Barais

Sunyé

et al. 2019

Software Testing Verif & Rel

View full text Add to dashboard Cite

SUMMARY Generative software development has paved the way for the creation of multiple code generators that serve as a basis for automatically generating code to different software and hardware platforms. In this context, the software quality becomes highly correlated to the quality of code generators used during software development. Eventual failures may result in a loss of confidence for the developers, who will unlikely continue to use these generators. It is then crucial to verify the correct behaviour of code generators in order to preserve software quality and reliability. In this paper, we leverage the metamorphic testing approach to automatically detect inconsistencies in code generators via so‐called “metamorphic relations”. We define the metamorphic relation (i.e., test oracle) as a comparison between the variations of performance and resource usage of test suites running on different versions of generated code. We rely on statistical methods to find the threshold value from which an unexpected variation is detected. We evaluate our approach by testing a family of code generators with respect to resource usage and performance metrics for five different target software platforms. The experimental results show that our approach is able to detect, among 95 executed test suites, 11 performance and 15 memory usage inconsistencies.

show abstract

“…In particular, Zhou and Sun combined MT and fuzzing and detected previously unknown fatal software faults in the LiDAR obstacle-perception module of the real-life selfdriving car system Baidu Apollo and reported the alarming results eight days before Uber's deadly crash in Tempe, AZ, USA, in March 2018 [20]. Also recently, Segura et al [21], [22] suggested that MT could be used to reveal performance failures. In the present study, we apply MT to the Adobe Experience Platform Launch Tag Manager software in the context of performance testing.…”

Section: Metamorphic Testingmentioning

confidence: 99%

Metamorphic Relations for Detection of Performance Anomalies

Johnston

Jarman

Berry

et al. 2019

2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET)

View full text Add to dashboard Cite

Metamorphic relations can be used to improve performance testing by comparing successive runs of the software under test. We examine one such metamorphic relation for page load times, which we used to discover and repair a race condition in the Adobe Experience Platform Launch Tag Manager. Histograms for page load times had different modalities, which alerted us to the presence of the bug. We discuss the need for performance measures in addition to the popular mean and standard deviation. We describe two approaches to automatically determine modality: Gaussian Mixture Models and the Silverman Test for Multimodality. Metamorphic relations which involve these performance measures can be used to alert engineers to the presence of performance anomalies.

show abstract

Performance Metamorphic Testing: Motivation and Challenges

Cited by 26 publications

References 7 publications

Empirical Assessment of Multimorphic Testing

Empirical Assessment of Multimorphic Testing

Leveraging metamorphic testing to automatically detect inconsistencies in code generator families

Metamorphic Relations for Detection of Performance Anomalies

Contact Info

Product

Resources

About