Designing for Software Testability Using Automated Oracles

Abstract

Software testing often requires massive numbers of test cases that must be manually inspected for correctness. This paper demonstrates the use of software "test oracles" to automate the process of checking the correctness of program output. The Prosper system, implemented by the authors, can be used to define test oracles and monitor the runtime behavior of software. An effective method to design software for testability must include the concurrent development of test oracles.

Identifying error proneness in path strata with genetic algorithms

“…Several methods for measuring software reliability are commonly used. They include software reliability models [1], automated oracles [2] and fault detection [3]. Malaiya et al [4], proposed a link between the levels of testing coverage (that is, the expectation to detect as many errors as possible with a test case) and the reliability of software.…”

Optimizing testing efficiency with error-prone path identification and genetic algorithms

“…Several methods for measuring software reliability are commonly used. They include software reliability models [1], automated oracles [2] and fault detection [3]. Malaiya et al [4], proposed a link between the levels of testing coverage (that is, the expectation to detect as many errors as possible with a test case) and the reliability of software.…”

Test orales for systems with complex outputs: the case of TTS systems

“…New software testing activities incorporate innovations in order to follow trends such as Object-Oriented (OO) , Aspect-Oriented (AO) (FERRARI;RASHID, 2008;CAFEO;MASIERO, 2011;KUMAR;SHARMA;GARG, 2009), web applications (PRAPHA-MONTRIPONG, 2012;MCMASTER;YUAN, 2012;LEOTTA et al, 2013), and embedded systems (YIN;LIU;NI, 2012;LATOUI;DJAHLI, 2013). This would allow one to affirm that test activities are highly dependent on the SUT.…”

“…According to (IVORY;HEARST, 2001), test oracles must support testers' decision about the SUT behavior against an input, establishing which results/behaviors are acceptable. In this sense, test oracles are reliable sources of informa-tion that testers can trust to decide the correct test results (MACHADO, 2000;TAKAHASHI, 2001;YIN, 1992). In technical terms, a test oracle can be carried out in different ways: functions, assertions, processes, data, parallel programs, and others (HARMAN et al, 2013).…”