Support planning and controlling of early quality assurance by combining expert judgment and defect data—a case study

Abstract

Planning quality assurance (QA) activities in a systematic way and controlling their execution are challenging tasks for companies that develop software or softwareintensive systems. Both require estimation capabilities regarding the effectiveness of the applied QA techniques and the defect content of the checked artifacts. Existing approaches for these purposes need extensive measurement data from historical projects. Due to the fact that many companies do not collect enough data for applying these approaches (especially for the early project lifecycle), they typically base their QA planning and controlling solely on expert opinion. This article presents a hybrid method combining commonly available measurement data and context-specific expert knowledge. To evaluate the method's applicability and usefulness, we conducted a case study in the context of independent verification and validation activities for critical software in the space domain. A hybrid defect content and effectiveness model was developed for the software requirements analysis phase and evaluated with available legacy data. One major result is that the hybrid model provides improved estimation accuracy when compared to applicable models based solely on data. The mean magnitude of relative error (MMRE) determined by crossvalidation is 29.6% compared to 76.5% obtained by the most accurate data-based model.

Aggregating predictions from experts: A review of statistical methods, experiments, and applications

“…Forecasting presents decision makers with actionable information that they can use to prevent (or prepare for) economic (Huang, Qiao, Wang, & Liu, 2016; Mak, Bui, & Blanning, 1996; Shin, Coh, & Lee, 2013), engineering (Guangliang, 1996; Neves & Frangopol, 2008; Zio, 1996), ecological (Borsuk, 2004; Failing, Horn, & Higgins, 2004; Johnson, Alhainen, Fox, Madsen, & Guillemain, 2018; Morales‐Nápoles, Paprotny, Worm, Abspoel‐Bukman, & Courage, 2017), social (Cabello, Conde, Diego, Moguerza, & Redchuk, 2012; Craig, Goldstein, Rougier, & Seheult, 2001; Kläs, Nakao, Elberzhager, & Münch, 2010), and public health burdens (Alho, 1992; Evans et al, 1994).…”

Software Evolution Prediction Using Seasonal Time Analysis: A Comparative Study

“…Kläs et al combined the use of time series with expert opinion to create prediction models for defects [4], to mitigate for the lack of historical information early in the development process. They reduced the mean magnitude of relative error from 76,5% (with a data-based model) to 29,6% (with their hybrid model).…”

Transparent combination of expert and measurement data for defect prediction

“…The model focuses on the overall development process with typical stages and QA activities, providing estimates for the defects introduced during development and the final number of residual defects. In order to support the planning of a specific QA activity, the model's abstraction level is seen as too high, especially because the set of predefined defect introduction factors may not be appropriate in specific contexts [14] and only a single factor in the model rates the effectiveness of a specific QA activity (with a five-level scale from "very low" to "extra high").…”

“…Dependant on the base effectiveness determined for the context and the EIF probability distribution computed for a project, if we had applied the sampling algorithms used in [14] and [15] in a straightforward manner, we could have obtained a positive probability for an effectiveness value greater than 100% (i.e., predicting a certain probability of finding more defects than the product contains). If we allowed such predictions, this would contradict reality.…”