Profile-Based, Load-Independent Anomaly Detection and Analysis in Performance Regression Testing of Software Systems

Ghaith, Shadi; Wang, Miao; Perry, Philip; Murphy, John

doi:10.1109/csmr.2013.54

Cited by 22 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then they compare the pairwise correlations between the vectors in the target run with those of the baseline run. Ghaith et al [32] proposed the use of queuing networks to detect performance regression. Jiang et al [5,33] introduced approaches to automatically detect anomaly in a performance test.…”

Section: Related Workmentioning

confidence: 99%

An industrial case study of automatically identifying performance regression-causes

Nguyen

Nagappan

Hassan

et al. 2014

Proceedings of the 11th Working Conference on Mining Software Repositories

View full text Add to dashboard Cite

Even the addition of a single extra field or control statement in the source code of a large-scale software system can lead to performance regressions. Such regressions can considerably degrade the user experience. Working closely with the members of a performance engineering team, we observe that they face a major challenge in identifying the cause of a performance regression given the large number of performance counters (e.g., memory and CPU usage) that must be analyzed. We propose the mining of a regression-causes repository (where the results of performance tests and causes of past regressions are stored) to assist the performance team in identifying the regression-cause of a newly-identified regression. We evaluate our approach on an open-source system, and a commercial system for which the team is responsible. The results show that our approach can accurately (up to 80% accuracy) identify performance regression-causes using a reasonably small number of historical test runs (sometimes as few as four test runs per regression-cause).

show abstract

Section: Related Workmentioning

confidence: 99%

An industrial case study of automatically identifying performance regression-causes

Nguyen

Nagappan

Hassan

et al. 2014

Proceedings of the 11th Working Conference on Mining Software Repositories

View full text Add to dashboard Cite

show abstract

“…The "no-worse-than-before" principle states that the average response time (system performance requirements) for the current version should be at least as good as prior versions [26]. [15], [38], [39], [40], [41], [42], [135], [174], [173], [177], [178] Comparing Against Derived (Threshold and/or target) Data Number of pass/fail requests, past performance metrics Detecting violations in performance and reliability requirements [15], [16], [26], [179] Detecting [86], [181], [182], [183], [184], [185], [186], [187], [188], [ Potential problematic log lines causing memory-related problems [190] • Deriving Target Data…”

Section: Comparing Against Derived Datamentioning

confidence: 99%

A Survey on Load Testing of Large-Scale Software Systems

Jiang

Hassan²

2015

IIEEE Trans. Software Eng.

136

View full text Add to dashboard Cite

Many large-scale software systems must service thousands or millions of concurrent requests. These systems must be load tested to ensure that they can function correctly under load (i.e., the rate of the incoming requests). In this paper, we survey the state of load testing research and practice. We compare and contrast current techniques that are used in the three phases of a load test: (1) designing a proper load, (2) executing a load test, and (3) analyzing the results of a load test. This survey will be useful for load testing practitioners and software engineering researchers with interest in the load testing of large-scale software systems.

show abstract

“…By this we extend on our previous work [9] where we suggested measuring the TP using a load generator software.…”

Section: Resources Utilization (Ru)mentioning

confidence: 96%

Automatic, load-independent detection of performance regressions by transaction profiles

Ghaith

Wang

Perry

et al. 2013

Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to Testing Automation

View full text Add to dashboard Cite

Performance regression testing is an important step in the production process of enterprise applications. Yet, analysing this type of testing data is mainly conducted manually and depends on the load applied during the test. To ease such a manual task we present an automated, load-independent technique to detect performance regression anomalies based on the analysis of performance testing data using a concept known as Transaction Profile. The approach can be automated and it utilises data already available to the performance testing along with the queueing network model of the testing system.The presented "Transaction Profile Run Report" was able to automatically catch performance regression anomalies caused by software changes and isolate them from those caused by load variations with a precision of 80% in a case study conducted against an open source application. Hence, by deploying our system, the testing teams are able to detect performance regression anomalies by avoiding the manual approach and eliminating the need to do extra runs with varying load.

show abstract

Profile-Based, Load-Independent Anomaly Detection and Analysis in Performance Regression Testing of Software Systems

Cited by 22 publications

References 11 publications

An industrial case study of automatically identifying performance regression-causes

An industrial case study of automatically identifying performance regression-causes

A Survey on Load Testing of Large-Scale Software Systems

Automatic, load-independent detection of performance regressions by transaction profiles

Contact Info

Product

Resources

About