A Self-assessment Instrument for Assessing Test Automation Maturity

Abstract

Päivi Raulamo-Jurvanen M3S, ITEE, Test automation is important in the software industry but selfassessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test automation; develop a self-assessment instrument (a survey) for assessing test automation maturity and scientifically evaluate it. We carried out the study in four stages. First, a literature review of 25 sources was conducted. Second, the initial instrument was developed. Third, seven experts from five companies evaluated the initial instrument. Content Validity Index and Cognitive Interview methods were used. Fourth, we revised the developed instrument. Our contributions are as follows: (a) we collected practices mapped into 15 key areas that indicate where an organization should focus to assess its test automation; (b) we developed and evaluated a self-assessment instrument for assessing test automation maturity; (c) we discuss important topics such as response bias that threatens self-assessment instruments. Our results help companies and researchers to understand and improve test automation practices and processes. CCS CONCEPTS• Software and its engineering → Software testing and debugging; Software development process management. Some test maturity models provide self-assessment instruments for organizations. For example, TPI provides a Test Maturity Matrix [34], TMap provides Checklists [1], TOM provides Test Organization Maturity Questionnaire [36]. The assessment items were defined in the form of survey questions. The assessment procedure is rather simple. Organizations answer assessment items (survey questions) to assess the maturity state of their test process [1,34,36].

Test automation maturity improves product quality—Quantitative study of open source projects using continuous integration

“…Our finding is consistent with findings made by prior scholars [27,35,28], who reported that better product quality can be achieved after adopting certain standard best practices for test automation maturity improvement in the CI context. However, our study examined a boarder range of standard best practices, as it was based on extensive literature reviews for standard best practices of test automation in our early studies [16,13].…”

“…Their findings were based on authors' experience. Many standard test automation best practices in the literature were not included, e.g., prioritizing important automated tests for execution, providing enough resources, and using the right test automation metrics to measure test automation performance [16]. To sum up, our empirical study can complement prior studies and make the novelty from the following aspects:…”

“…We reused our previous test automation maturity survey developed in our 2020 study [13]. That survey was developed using a knowledge base established upon 18 test maturity models [13,16]. It has been reviewed by test automation experts and executed with 151 practitioners (coming from above 101 organizations in 25 countries) in the industry.…”

“…Prior scholars described that, at the high level of test automation maturity, "test automation practices are defined, managed, measured, controlled, and effective within the organization" [15,13,6,12]. The existing literature has presented a set of standard best practices to guide organizations to reach the high level of test automation maturity [12,16,17]. For instance, in many test maturity models (such as TMap [18], TestSPICE [19], and TAIM [20]), an example best practice is -define a test automation strategy to conduct test automation activities under given boundary conditions.…”

Development of a valid and reliable software customization model for SaaS quality through iterative method: perspectives from academia

“…While designing the data analysis for each round, we primarily followed the content validity index (CVI) method, which is the common method for this purpose (Bhatti & Ahsan, 2017), as guided by (Polit & Beck, 2006). The popularity of CVI is not only limited to educational, psychological, or nursing research, but also to other disciplines or research areas, such as researches in software engineering and information systems (Bhatti & Ahsan, 2017;Yilmaz et al, 2017;Wang et al, 2019). In this study, two quantities were calculated (Polit & Beck, 2006): 1.…”