Testing scientific software: A systematic literature review

Kanewala, Upulee; Bieman, James M.

doi:10.1016/j.infsof.2014.05.006

Cited by 116 publications

(111 citation statements)

References 67 publications

Supporting

Mentioning

100

Contrasting

Unclassified

Order By: Relevance

“…A key challenge found by Kanewala and Bieman [2014] is the availability of a test oracle: a means of computing an expected result for comparison with the output from the software under development. Oracles come in a variety of forms, including manual computation, earlier prototypes and third party reference implementations [Weyuker 1982].…”

Section: Testingmentioning

confidence: 99%

Bridging the Chasm

Storer

2017

ACM Comput. Surv.

View full text Add to dashboard Cite

The use of software is pervasive in all fields of science. Associated software development efforts may be very large, long lived, and complex, requiring the commitment of significant resources. However, several authors have argued that the “gap” or “chasm” between software engineering and scientific programming is a serious risk to the production of reliable scientific results, as demonstrated in a number of case studies. This article reviews the research that addresses the gap, exploring how both software engineering and research practice may need to evolve to accommodate the use of software in science.

show abstract

Section: Testingmentioning

confidence: 99%

Bridging the Chasm

Storer

2017

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…One of the greatest challenges for testing scientific software is due to the oracle problem [8] [1]. Metamorphic testing is the technique for addressing the oracle problem though developing them with MRs [2] [16].…”

Section: Related Workmentioning

confidence: 99%

“…Scientific software is the software that includes computational components for supporting scientific investigation and decision making [8]. The examples of scientific software include simulation software of nuclear reactions, software for predicting and tracking hurricanes, and software for analyzing medical images.…”

Section: Introductionmentioning

confidence: 99%

“…Many of them are "non-testable" due to the absence of test oracles [2]. Oracle problems are key to solve for adequately testing scientific software [8]. Metamorphic testing [2] as a novel software testing technique is a promising approach for solving oracle problems.…”

Section: Introductionmentioning

confidence: 99%

“…ADDA is a fairly complex scientific software system with many characteristics that cause the testing challenges that were described by Kanewala and Bieman [8]. The most challenge issue for testing ADDA is the absence of test oracles since a test input of ADDA may include thousands of parameters and it is impossible to know the correctness of the output for an arbitrary input.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software

Ding¹,

Zhang²

2016

International Conferences on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

Abstract-Absence of test oracles is the grand challenge for testing complex scientific software. Metamorphic testing is the novel technique for developing test oracles on metamorphic relations. Although it is easy to find metamorphic relations based on general guidelines and domain knowledge, the ones that can adequately test the software are difficult to be developed. This paper introduces a machine learning approach for iteratively developing metamorphic relations. The approach develops initial metamorphic relations and tests first, and then the relations and tests are refined through mining the initial test execution and evaluation results with machine learning algorithms. The approach and its effectiveness are illustrated through testing an open source discrete dipole approximation program.

show abstract

Towards an AI‐driven business development framework: A multi‐case study

John

Olsson

Bosch

2022

J Software Evolu Process

View full text Add to dashboard Cite

Artificial intelligence (AI) and the use of machine learning (ML) and deep learning (DL) technologies are becoming increasingly popular in companies. These technologies enable companies to leverage big quantities of data to improve system performance and accelerate business development. However, despite the appeal of ML/DL, there is a lack of systematic and structured methods and processes to help data scientists and other company roles and functions to develop, deploy and evolve models. In this paper, based on multi-case study research in six companies, we explore practices and challenges practitioners experience in developing ML/DL models as part of large software-intensive embedded systems. Based on our empirical findings, we derive a conceptual framework in which we identify three high-level activities that companies perform in parallel with the development, deployment and evolution of models. Within this framework, we outline activities, iterations and triggers that optimize model design as well as roles and company functions. In this way, we provide practitioners with a blueprint for effectively integrating ML/DL model development into the business to achieve better results than other (algorithmic) approaches. In addition, we show how this framework helps companies solve the challenges we have identified and discuss checkpoints for terminating the business case.

show abstract

Testing scientific software: A systematic literature review

Cited by 116 publications

References 67 publications

Bridging the Chasm

Bridging the Chasm

A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software

Towards an AI‐driven business development framework: A multi‐case study

Contact Info

Product

Resources

About