Revisiting the problem of using problem reports for quality assessment

Abstract

In this paper, we describe our experience with using problem reports from industry for quality assessment. The non-uniform terminology used in problem reports and validity concerns have been subject of earlier research but are far from settled. To distinguish between terms such as defects or errors, we propose to answer three questions on the scope of a study related to what (problem appearance or its cause), where (problems related to software; executable or not; or system), and when (problems recorded in all development life cycles or some of them). Challenges in defining research questions and metrics, collecting and analyzing data, generalizing the results and reporting them are discussed. Ambiguity in defining problem report fields and missing, inconsistent or wrong data threatens the value of collected evidence. Some of these concerns could be settled by answering some basic questions related to the problem reporting fields and improving data collection routines and tools.

Component Testing Is Not Enough - A Study of Software Faults in Telecom Middleware

“…The related terminology in this area (fault, error, cause or reason, failure, bug, defect, and anomaly) is often confusing because these terms are used interchangeably and inconsistently by many in industry and academia; see further discussion in Mohaghegi et al [27]. Therefore we define the following terms with inspiration from earlier work from Avižienis & Laprie [10] and Thane [9], where a fault is the static origin in the code, that during dynamic execution propagates (in Figure 1 described as by a solid arrow) to an error (which is an intermediate infection of the code).…”

A Case Study of Source Code Evolution

“…Studies that rely on fault reports have expressed several validity concerns regarding the reliability of these reports as a source of data for empirical research. Several of these concerns, including subjectivity, inaccuracy, and ambiguity of the reports, have been discussed by Mohagheghi et al [14] and Ostrand et al [15]. The use of the history of the source code changes has increased the validity of the results obtained from the present study.…”

Improving IT Service Management Processes: A Case Study on IT Service Support

“…Previous studies have dealt with predicting incident volumes by statistical methods [8], service desk and incident management challenges [9], activity-based management of IT service delivery [10], knowledge management-centric help desk [11], the maturity of the problem management process [12], using problem reports for quality assessments [13], dynamic change management model for managing IT changes [14], release management in component-based development [15], open source software releases [16], critical elements of the patch management process [17], and patch management challenges [18].…”