Findings from Phase 2 of the SPICE trials

Abstract

The international SPICE (Software Process Improvement and Capability dEtermination) project was set up to support the development of the ISO/IEC 15504 standard for software process assessment (SPA). The project mounted a set of trials to validate the emerging standard against the goals and requirements defined at the start of the SPICE project and to verify the consistency and usability of its component parts. A considerable number of empirical evaluation studies have been conducted during the Phase 2 SPICE Trials based on ISO/IEC PDTR 15504 (between September 1996 and June 1998). Such an exercise is unprecedented in the software engineering standards community and it provides a unique opportunity for empirical validation. The purpose of this paper is to present major parts of the findings of the empirical studies conducted as part of the SPICE Project during Phase 2 of the SPICE Trials. The topics covered in this paper include (i) investigation into reasons for performing SPAs, (ii) evaluation of the internal consistency of the capability dimension, (iii) use of interrater agreement as a measure of the reliability of assessments, (iv) evaluation of the predictive validity of process capability, (v) evaluation of an exemplar model (Part 5), (vi) identification of factors influencing assessor effort, and (vii) empirical comparison between ISO/IEC PDTR 15504 and ISO 9001. Major lessons learned as well as future research directions are summarized on the strengths and weaknesses of ISO/IEC 15505. Copyright © 2001 John Wiley & Sons, Ltd.

Systematic review of organizational motivations for adopting CMM-based SPI

“…The 49 Studies covered a total of 348 organizations, but two Studies [6,12] did not say how many organizations Table 3 Data for each Publication as a Whole Attribute Description…”

“…Of the 49 Studies, 45 were case studies or experience reports about single organizations, one was a series of interviews [17] with 29 organizations, one was an industrial trial [12] of 44 organizations, one was a survey [11] of 85 organizations, and one was a survey [6] of 161 organizations. The interview study [17] covered organizations using CMM-based SPI as well as other forms of SPI, but on request the study's author (also an author of this systematic review) was able to supply previously unpublished data on only those organizations (13 of the 29) that used CMMbased SPI.…”

Software process improvement as emergent change: A structurational analysis

“…This similarity often means that organisations with ISO 9001 accreditation are assessed at above the base levels of the different maturity models (Paulk, 1993;Jung et al, 2001). The quality standard defines the minimum criteria for a quality management system, with the emphasis on documented processes (how to do it) and records of product quality (what was achieved).…”

“…The focus on improving the process emphasises the internal activity over the external competitive position. The primary reasons for organisations performing an assessment using SPICE (ISO 15504) reflect this internal focus, with little importance being placed upon the external perception or market 3() benefits (Jung et al 2001 ). In the SPICE trials approximately 60 percent did not observe a major impact on the organisation (reported in ('onradi and Fuggetta, 2002).…”

The internal consistency and precedence of key process areas in the capability maturity model for software

“…It is crucial to estimate the amount of error in order to give confidence in assessment results. El-Emam et al (1998) and Jung et al (2001) also show that more reliable assessments can reduce effort during appraisal team consensus to determine the validity of findings and establish ratings. In addition, questionnaire-based studies of SW-CMM assessment team leaders and team members indicate that consensus building is one of the most difficult aspects of assessment (Dunaway and Baker 2001).…”