Automating image segmentation verification and validation by learning test oracles

Frounchi, Kambiz; Briand, Lionel C.; Grady, Leo; Labiche, Yvan; Subramanyan, Rajesh

doi:10.1016/j.infsof.2011.06.009

Cited by 28 publications

(25 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Datasets can also vary in terms of size. Larger datasets have less bias ) and are less susceptible to the negative effects of noise (Frounchi et al 2011).…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

“…Several trends between the number of feature extractors used and effectiveness can be observed: improvement, stagnation, and decline. Two of the feature extractors used in a study conducted by Frounchi et al (2011) include the Tanimoto Coefficient T C and Scalable ODI SODI . In this study, it was observed that one set of features that consisted of {T C} was the most effective set for negative classifications, and that another set of feature extractors that contained {T C, SODI } was the most effective set for positive classifications.…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

“…Using multiple members from G is unlikely to significantly improve classification accuracy (Frounchi et al 2011). This could explain stagnation trends.…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

“…Different feature extractors may have different quality attributes, e.g. levels of efficiency (Frounchi et al 2011) or generalisability and so some may be more favourable than others in certain contexts. Thus, one may consider choosing a subset of G based on the quality attributes offered by the different feature extractors in G. Techniques like wrappers and filters can identify and remove feature extractors that will not significantly improve classification accuracy (Frounchi et al 2011) and thus can purge excess members of such a group.…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

“…However, it has been reported that the choice of algorithm does not have a significant impact on effectiveness, and thus, these algorithms might be interchangeable (Frounchi et al 2011).…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

See 4 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

The terms "Oracle Problem" and "Non-testable system" interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in these programs. The prevalence of the oracle problem has inspired the research community to develop several automated testing techniques that can detect functional software faults in such programs. These techniques include N-Version testing, Metamorphic Testing, Assertions, Machine Learning Oracles, and Statistical Hypothesis Testing. This paper presents a Mapping Study that covers these techniques. The Mapping Study presents a series of discussions about each technique, from different perspectives, e.g. effectiveness, efficiency, and usability. It also presents a comparative analysis of these techniques in terms of these perspectives. Finally, potential research opportunities within the non-testable systems problem domain are highlighted within the Mapping Study. We believe that the aforementioned discussions and comparative analysis will be invaluable for new researchers that are attempting to familiarise themselves with the field, and be a useful resource for practitioners that are in the process of selecting an appropriate technique for their context, or deciding how to apply their selected technique. We also believe that our own insights, which are embedded throughout these discussions and the comparative analysis, will be useful for researchers that are already accustomed to the field. It is our hope that the potential research opportunities that have been highlighted by the Mapping Study will steer the direction of future research endeavours.

show abstract

“…Datasets can also vary in terms of size. Larger datasets have less bias ) and are less susceptible to the negative effects of noise (Frounchi et al 2011).…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

“…Using multiple members from G is unlikely to significantly improve classification accuracy (Frounchi et al 2011). This could explain stagnation trends.…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

“…However, it has been reported that the choice of algorithm does not have a significant impact on effectiveness, and thus, these algorithms might be interchangeable (Frounchi et al 2011).…”

Section: Design and Application Of Machine Learning Oraclesmentioning

confidence: 99%

See 3 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

show abstract

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Kanewala

Bieman

Ben‐Hur

2015

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Comprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behaviour of the programme. But the challenges in creating suitable oracles limit the ability to perform automated testing in some programmes, and especially in scientific software. Metamorphic testing is a method for automating the testing process for programmes without test oracles. This technique operates by checking whether the programme behaves according to properties called metamorphic relations. A metamorphic relation describes the change in output when the input is changed in a prescribed way. Unfortunately, finding the metamorphic relations satisfied by a programme or function remains a labour‐intensive task, which is generally performed by a domain expert or a programmer. In this work, we propose a machine learning approach for predicting metamorphic relations that uses a graph‐based representation of a programme to represent control flow and data dependency information. In earlier work, we found that simple features derived from such graphs provide good performance. An analysis of the features used in this earlier work led us to explore the effectiveness of several representations of those graphs using the machine learning framework of graph kernels, which provide various ways of measuring similarity between graphs. Our results show that a graph kernel that evaluates the contribution of all paths in the graph has the best accuracy and that control flow information is more useful than data dependency information. The data used in this study are available for download at http://www.cs.colostate.edu/saxs/MRpred/functions.tar.gz to help researchers in further development of metamorphic relation prediction methods. Copyright © 2015 John Wiley & Sons, Ltd.

show abstract