Using goal–question–metric to compare research and practice perspectives on regression testing

Abstract

Regression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large-scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal-question-metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.