The Value of Certifying Software Release Readiness: An Exploratory Study of Certification for a Critical System at JPL

Abstract

A software release is a decision to deliver code to an organization outside of the development team usually for testing or operational purposes. For critical systems this can be a risky decision where failure to pass a test or holding up the project schedule can have a major impact. The release decision is primarily based on the understanding on the level of quality the software currently has (be it high quality, low quality, or unknown). But for large, complex systems, determining the level of quality with high confidence is a challenge. A poor understanding of the confidence in the quality level increases decision risk leading potentially to a bad release decision that possibly could have been avoided had the confidence in the quality been better known. Certification of release readiness attempts to address this risk by building confidence in the quality level. But this comes at a cost and the relationship between certification and decision risk reduction has not been well understood. This work describes our experience investigating the value of certification and our efforts to improve the mandated software readiness certification record (SRCR) process. A well known critical system at JPL is used as a case study to exemplify this effort.

FactSheets: Increasing trust in AI services through supplier's declarations of conformity

“…They predict the performance of the software on unseen inputs using the MTTF (mean time to failure) metric. Port and Wilf [28] describe a procedure to certify the readiness for software release, understanding the tradeoff in cost of too early a release due to failures in the field, versus the cost in personnel and schedule delay arising from more extensive testing. Their technique involves the filling out of a questionnaire by the software developer called the Software Review and Certification Record (SRCR), which is 'credentialed' with signatories who approve the document prior to the release decision.…”

Optimized Order of Software Testing Techniques in Agile Process – A Systematic Approach

“…The goal of release readiness review is to assess the quality of the product with reference to any risks involved in delivery of product [8].…”

“…In [8] the idea of Release Readiness Review is to certify a combination of checks necessary before rolling out a software release. In agile methodology, a software product is assessed with respect to RRR document at the end of final iteration.…”

“…The research work [8], proposes a checklist for evaluating all the mandatory and relevant aspects for releasing a quality product and concerning responsible authorities sign off the checklist. This paper proposes an optimized combination of testing strategies considering the appropriate techniques at right stage of the agile methodology for developing and delivering a quality product.…”

Evaluation of Software Degradation and Forecasting Future Development Needs in Software Evolution

“…According to [10], almost 50% of software life cycle cost is attributed to maintenance; and yet, relatively very little is known about the software maintenance process and the factors that influence its cost. With regards to release readiness, [11] opined that, a poor understanding of the confidence in the quality level increases decision risk leading potentially to a bad release decision that possibly could have been avoided had the confidence in the quality been better known. A well-known critical system at jpl was used as a case study to investigate the value of certification to improve the mandated software readiness certification record (srcr) process.…”