A case study on testing CMM uncertainty simulation software (VCMM)

Abackerli, Alvaro José; Pereira, Paulo Henrique Cruz; Calonego, Nivaldi

doi:10.1590/s1678-58782010000100002

Cited by 10 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RQ2: Are there special characteristics or faults in scientific software or its development that make testing difficult? We identified two categories of challenges in scientific software testing: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and model as inseparable entities.…”

Section: Discussionmentioning

confidence: 99%

“…Using analytical solutions is sometimes preferred over pseudo oracles since they can identify common algorithmic errors among the implementations. For example, a theoretically calculated rate of convergence can be compared with the rate produced by the code to check for faults in the program [1,38,24]. Limitations: Analytical solutions may not be available for every application [9] and may not be accurate due to human errors [65].…”

Section: Rq3: Can We Use Existing Testing Methods (Or Adapt Them) To mentioning

confidence: 99%

“…Software faults such as one-off errors have caused the loss of precision in seismic data processing programs [27]. Software faults have compromised coordinate measuring machine (CMM) performance [1]. In addition, scientists have been forced to retract published work due to software faults [51].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Testing scientific software: A systematic literature review

Kanewala

Bieman

2014

Information and Software Technology

116

100

View full text Add to dashboard Cite

Context Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. Objective This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. Method We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software. Results We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them. Conclusions Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Rq3: Can We Use Existing Testing Methods (Or Adapt Them) To mentioning

confidence: 99%

See 1 more Smart Citation

Testing scientific software: A systematic literature review

Kanewala

Bieman

2014

Information and Software Technology

116

100

View full text Add to dashboard Cite

show abstract

“…These subtle faults can cause incorrect programme output without causing a programme to crash. For example, one‐off errors caused loss of precision in seismic data processing programmes , and software faults compromised the performance of coordinate measuring machines . In addition, there have been multiple retractions of published work because of software faults .…”

Section: Introductionmentioning

confidence: 99%

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Kanewala

Bieman

Ben‐Hur

2015

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Comprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behaviour of the programme. But the challenges in creating suitable oracles limit the ability to perform automated testing in some programmes, and especially in scientific software. Metamorphic testing is a method for automating the testing process for programmes without test oracles. This technique operates by checking whether the programme behaves according to properties called metamorphic relations. A metamorphic relation describes the change in output when the input is changed in a prescribed way. Unfortunately, finding the metamorphic relations satisfied by a programme or function remains a labour‐intensive task, which is generally performed by a domain expert or a programmer. In this work, we propose a machine learning approach for predicting metamorphic relations that uses a graph‐based representation of a programme to represent control flow and data dependency information. In earlier work, we found that simple features derived from such graphs provide good performance. An analysis of the features used in this earlier work led us to explore the effectiveness of several representations of those graphs using the machine learning framework of graph kernels, which provide various ways of measuring similarity between graphs. Our results show that a graph kernel that evaluates the contribution of all paths in the graph has the best accuracy and that control flow information is more useful than data dependency information. The data used in this study are available for download at http://www.cs.colostate.edu/saxs/MRpred/functions.tar.gz to help researchers in further development of metamorphic relation prediction methods. Copyright © 2015 John Wiley & Sons, Ltd.

show abstract

“…The whole equipment evaluation is essential to ensure its proper operation. The increasing use of computers in measurements has highlighted the need to verify the software as an integral part of measurement systems [14]. According to [15], it is as important to declare the measurement uncertainty as it is to report the measurement result itself.…”

Section: Introductionmentioning

confidence: 99%

Traceability for measurements carried out on incremental step loading equipment

Júnior

Leal

Pires

et al. 2019

J Braz. Soc. Mech. Sci. Eng.

View full text Add to dashboard Cite

The incremental step loading technique has been widely used for material hydrogen embrittlement characterization. However, due to numerous error sources that can affect the measurement results and to the current lack of standardization, users are facing difficulties in the evaluation of measurement uncertainty as well as in the metrological performance evaluation of incremental step loading equipment. The paper deals with evaluation of the metrological performance of equipment used to characterize the susceptibility of materials to hydrogen embrittlement through use of incremental step loading technique. Additionally, a procedure is developed to estimate the measurement uncertainty associated with the results obtained during the test. Two equipment configurations were verified. The first one, with a load cell of 20 kN, was used for test specimens with nominal dimensions of 30 mm, 30 mm and 200 mm in width, thickness and length, respectively; and the second, with a 1 kN load cell was used for test specimens with nominal dimensions of 10 mm × 10 mm × 60 mm. Three measurands were evaluated in each assemblage (force applied on the specimen, equipment arm positioning angle and stress). Several metrological parameters were estimated (bias, accuracy, expanded uncertainty, repeatability, maximum error, hysteresis and nonlinearity). The effectiveness of the proposed procedure for uncertainty assessment was tested by calculation of the uncertainty associated with lower value for fast fracture strength. From the obtained results, it was concluded that the two equipment configurations had excellent metrological properties, in both the loading and unloading phases. It was shown that the proposed procedure can properly estimate the uncertainty associated with measurement on incremental step loading testing. Considering the widely use of the incremental step loading technique, the results here presented can be particularly useful for ensuring the traceability of results to the International System of Units required by the ISO 17025 standard.

show abstract

A case study on testing CMM uncertainty simulation software (VCMM)

Abstract: A Case Study on Testing CMM Uncertainty Simulation Software (VCMM)Virtual

Cited by 10 publications

References 21 publications

Testing scientific software: A systematic literature review

Testing scientific software: A systematic literature review

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Traceability for measurements carried out on incremental step loading equipment

Contact Info

Product

Resources

About