Regression testing for large-scale embedded software development – Exploring the state of practice

Abstract

Context. A majority of the regression testing techniques proposed by the research have not been adopted in industry. To increase adoption rates, we need to better understand the practitioners' perspectives on regression testing. Objective. This study aims at exploring the regression testing state of practice in the large-scale embedded software development. The study has two objectives, 1) to highlight the potential challenges in practice, and 2) to identify the industry-relevant research areas regarding regression testing. Method. We conducted a qualitative study in two large-scale embedded software development companies, where we carried out semi-structured interviews with representatives from five software testing teams. We did conduct the detailed review of the process documentation of the companies to complement/validate the findings of the interviews. Results. Mostly, the practitioners run regression testing with a selected scope, the selection of scope depends upon the size, complexity, and location of the change. Test cases are prioritized on the basis of risk and critical functionality. The practitioners rely on their knowledge and experience for the decision making regarding selection and prioritization of test cases. The companies are using both automated and manual regression testing, and mainly they rely on inhouse developed tools for test automation. The challenges identified in the companies are: time to test, information management, test suite maintenance, lack of communication, test selection/prioritization, lack of assessment, etc. The proposed improvements are in line with the identified challenges. Regression testing goals identified in this study are customer satisfaction, critical defect detection, confidence, effectiveness, efficiency, and controlled slip through of faults. Conclusions. Considering the current state of practice and identified challenges we conclude that there is a need to reconsider the regression test strategy in the Preprint submitted to Information and Software Technology November 29, 2018 companies. Researchers need to analyze the industry perspective while proposing new regression testing techniques. The industry-academia collaboration projects would be a good platform in this regard.

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

“…In another study, the industrial collaborators consider fault probability, time and cost associated to test case setup and execution, and requirement coverage as the most relevant information [104]. Change-related attributes were identified as the primary information for testing an embedded system [76].…”

“…Under this problem category, one important aspect is whether the information attributes remain static or have a dynamic or evolving nature. In the latter case, the cost-benefit of implementing a specific TCP method will depend on how frequently the information is expected to change and how often TCP is executed [55,76].…”

“…The adoption of RT also presents severe limitations if the information sources required by a particular technique are unavailable [2] or are not aligned with the company needs. Furthermore, the relevance of the information attributes might depend on organisational practices and projectspecific factors [52,76]. Engström et al conclude that a history-based approach better fits the interests of their industrial partner, since detection effectiveness, test case age and time since last execution are highly significant [36].…”

“…Taxonomies like SERP (Software Engineering Research and Practice) are also a good instrument to support academia-industry communication [35], clearly defining the problem under study and mapping challenges to solutions [89]. In fact, the lack of communication and differences in terminology have been identified as key factors hampering the transfer of RT research to industry [76]. In the context of software testing, SERP-test -an adaptation of SERP to this area-allows structuring software testing research with respect to testing tasks, context constraints and expected effects.…”

“…As the software evolves, regression testing (RT) activities are planned to guarantee that changes in the system do not alter the behaviour of stable parts [120]. In modern scenarios of continuous integration and delivery, re-running the whole test suite for large-scale industrial systems is impractical [76,117]. Therefore, selecting and prioritising the most relevant test cases from extensive test suites becomes a core step of RT [65].…”