The Oracle Problem in Software Testing: A Survey

Barr, Earl T.; Harman, Mark; McMinn, Phil; Shahbaz, Muzammil; Yoo, Shin

doi:10.1109/tse.2014.2372785

Cited by 692 publications

(377 citation statements)

References 183 publications

(201 reference statements)

Supporting

Mentioning

368

Contrasting

Unclassified

Order By: Relevance

“…Three extensive reviews of test oracles exist: by Baresi and Young (2001), by Pezzè and Zhang (2005), and by Barr et al (2015) who classified the existing literature on test oracles into three broad categories:-specified oracles; implicit oracles; and derived oracles. Specified oracles are test oracles obtained from formal specification of the system behaviour.…”

Section: Background and Related Workmentioning

confidence: 99%

Separating passing and failing test executions by clustering anomalies

Almaghairbe

Roper

2016

Software Qual J

View full text Add to dashboard Cite

Developments in the automation of test data generation have greatly improved efficiency of the software testing process, but the so-called oracle problem (deciding the pass or fail outcome of a test execution) is still primarily an expensive and error-prone manual activity. We present an approach to automatically detect passing and failing executions using cluster-based anomaly detection on dynamic execution data based on firstly, just a system's input/output pairs and secondly, amalgamations of input/output pairs and execution traces. The key hypothesis is that failures will group into small clusters, whereas passing executions will group into larger ones. Evaluation on three systems with a range of faults demonstrates this hypothesis to be valid-in many cases small clusters were composed of at least 60 % failures (and often more). Concentrating the failures in these small clusters substantially reduces the numbers of outputs that a developer would need to manually examine following a test run and illustrates that the approach has the potential to improve the effectiveness and efficiency of the testing process.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Separating passing and failing test executions by clustering anomalies

Almaghairbe

Roper

2016

Software Qual J

View full text Add to dashboard Cite

show abstract

“…The idea is to use a model of the system, it can be created in different ways.. Indeed, this problem is the more important problem of system test automation [23]. Specifically, in our field, a model can be generated by several ways.…”

Section: Smart Card Application Assurancementioning

confidence: 99%

Generation of Local and Expected Behaviors of a Smart Card Application to Detect Software Anomaly

Jolly

Hemery

Rosenberger

2015

2015 10th International Conference on Availability, Reliability and Security

View full text Add to dashboard Cite

Abstract-The electronic payment transaction involves the use of a smart card. A card application is a software, corresponding to standards and non-proprietary and proprietary specifications, and is stored in the smart card. Despite increased security with Europay Mastercard Visa (EMV) specifications, attacks still exist due to anomalies in the card application. The validation of the card application enables the detection of any anomaly, improving the overall security of electronic payment transactions. Among the different ways of validating a card application, we can use the verification of required behaviors. These behavior can be materialized as properties of commands sent by the terminal and responses from the smart card, using the Application Protocol Data Unit (APDU) from the ISO/IEC 7816 standard [1]. However, the creation of these behaviors is complicated. We propose in this article a way to automatically create such behaviors by using a genetic algorithm technique.

show abstract

“…They generate test cases for a given unit to test, aiming at maximizing one or more coverage criteria; such an unit is typically a class under test (CUT). However, those tests still need human intervention: human effort in reading, understanding and modifying test cases is necessary, especially since the oracles have to be manually specified [2].…”

Section: Introductionmentioning

confidence: 99%

An empirical investigation on the readability of manual and generated test cases

Grano

Scalabrino

Gall

et al. 2018

Proceedings of the 26th Conference on Program Comprehension

View full text Add to dashboard Cite

Software testing is one of the most crucial tasks in the typical development process. Developers are usually required to write unit test cases for the code they implement. Since this is a time-consuming task, in last years many approaches and tools for automatic test case generation -such as EvoSuitehave been introduced. Nevertheless, developers have to maintain and evolve tests to sustain the changes in the source code; therefore, having readable test cases is important to ease such a process. However, it is still not clear whether developers make an effort in writing readable unit tests. Therefore, in this paper, we conduct an explorative study comparing the readability of manually written test cases with the classes they test. Moreover, we deepen such analysis looking at the readability of automatically generated test cases. Our results suggest that developers tend to neglect the readability of test cases and that automatically generated test cases are generally even less readable than manually written ones. ABSTRACTSoftware testing is one of the most crucial tasks in the typical development process. Developers are usually required to write unit test cases for the code they implement. Since this is a time-consuming task, in last years many approaches and tools for automatic test case generation -such as EvoSuite-have been introduced. Nevertheless, developers have to maintain and evolve tests to sustain the changes in the source code; therefore, having readable test cases is important to ease such a process. However, it is still not clear whether developers make an effort in writing readable unit tests. Therefore, in this paper, we conduct an explorative study comparing the readability of manually written test cases with the classes they test. Moreover, we deepen such analysis looking at the readability of automatically generated test cases. Our results suggest that developers tend to neglect the readability of test cases and that automatically generated test cases are generally even less readable than manually written ones. CCS CONCEPTS• Software and its engineering → Maintaining software; Search-based software engineering;

show abstract

The Oracle Problem in Software Testing: A Survey

Cited by 692 publications

References 183 publications

Separating passing and failing test executions by clustering anomalies

Separating passing and failing test executions by clustering anomalies

Generation of Local and Expected Behaviors of a Smart Card Application to Detect Software Anomaly

An empirical investigation on the readability of manual and generated test cases

Contact Info

Product

Resources

About