An orchestrated survey of methodologies for automated software test case generation

Anand, Sandeep; Burke, Edmund K.; Chen, Tsong Yueh; Clark, John A.; Cohen, Myra B.; Grieskamp, Wolfgang; Harman, Mark; Harrold, Mary Jean; McMinn, Phil; Bertolino, Antonia; Li, J. Jenny; Zhu, Hong

doi:10.1016/j.jss.2013.02.061

Cited by 527 publications

(367 citation statements)

References 189 publications

Supporting

Mentioning

361

Contrasting

Unclassified

Order By: Relevance

“…In the field of software testing, there is a family of test case generation methods, known as adaptive random testing (ART), which is designed to improve the fault-detection effectiveness of random testing by enforcing an even spread of randomly generated test cases over the input domain [2,3]. ART is based on the observation that failure-causing inputs tend to cluster, forming contiguous failure regions.…”

Section: Related Workmentioning

confidence: 99%

“…ART well preserves the randomness of test cases, and outperforms RT in terms of both F-measure and P-measure [2]. Naturally, however, generating an adaptive random test case using the location information of already executed test cases do require more computational overhead as compared with the generation of a pure random number.…”

Section: Related Workmentioning

confidence: 99%

“…Let S=(s 1 , s 2 , …, s n ) be sequence (i) as defined in Section I, that is, S is a sequence of n consecutively generated states; let T=(t 1 , t 2 , …, t n ) be an adaptive random sequence of integers in the range of [0, MAX], where MAX is the maximum integer of the system and MAX > n. We say t i is an adaptive random number, i = 1, 2, …, n. Most of the ART family of algorithms [2,3] can be used by SBS to generate T, and SBS sets the priority number of state s i to t i , i=1, 2, …, n. When the next state s n+1 is generated, SBS will call the adaptive random number generator to generate the next adaptive random number t n+1 , which will serve as the priority number of s n+1 . In this way, the generated states are checked for property violations following the descending order of their priority numbers, and state generation and state verification can be interleaved.…”

Section: An Algorithm Of Generating Priority Numbersmentioning

confidence: 99%

See 2 more Smart Citations

Similarity-Based Search for Model Checking: A Pilot Study with Java PathFinder

Ibrahimov

Wang

Zhi

2013

2013 13th International Conference on Quality Software

View full text Add to dashboard Cite

When a model checker cannot explore the entire state space because of limited resources, model checking becomes a kind of testing with an attempt to find a failure (violation of properties) quickly. We consider two state sequences in model checking: (i) the sequence in which new states are generated, and (ii) the sequence in which the states generated in sequence (i) are checked for property violation. We observe that neighboring states in sequence (i) often have similarities in certain ways. Based on this observation we propose a search strategy, which generates sequence (ii) in such a way that similar states are evenly spread over the sequence. As a result, neighboring states in sequence (ii) can have a higher diversity. A pilot empirical study with Java PathFinder suggests that the proposed strategy can outperform random search in terms of creating equal or smaller number of states to detect a failure. Abstract --When a model checker cannot explore the entire state space because of limited resources, model checking becomes a kind of testing with an attempt to find a failure (violation of properties) quickly. We consider two state sequences in model checking: (i) the sequence in which new states are generated, and (ii) the sequence in which the states generated in sequence (i) are checked for property violation. We observe that neighboring states in sequence (i) often have similarities in certain ways. Based on this observation we propose a search strategy, which generates sequence (ii) in such a way that similar states are evenly spread over the sequence. As a result, neighboring states in sequence (ii) can have a higher diversity. A pilot empirical study with Java PathFinder suggests that the proposed strategy can outperform random search in terms of creating equal or smaller number of states to detect a failure.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: An Algorithm Of Generating Priority Numbersmentioning

confidence: 99%

See 1 more Smart Citation

Similarity-Based Search for Model Checking: A Pilot Study with Java PathFinder

Ibrahimov

Wang

Zhi

2013

2013 13th International Conference on Quality Software

View full text Add to dashboard Cite

show abstract

“…According to an analysis performed over a sample of 100 open source projects called SF100 [8] 1 , we discovered that almost one third of the methods have at least one EDP [7]. Although the number of EDPs of a program is high, and exception handling is an important topic in software development [4], the influence of exception mechanisms to unit test data generation using symbolic execution has not been widely explored [1,3,5].Taking into account the implicit constraints that stem from exception handling mechanisms can significantly increase the number of path constraints. This has the potential to exacerbate a well-known issue faced by symbolic execution approaches: path explosion [1,5,6], which is usually caused by complex loop structures.…”

mentioning

confidence: 99%

Analyzing Exceptions in the Context of Test Data Generation Based on Symbolic Execution

Eler

Durelli

Endo

2015

International Conferences on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

Abstract-Testing exception scenarios is a challenging task in the context of test data generation based on symbolic execution. In such a context, test data is generated based on constraints explicitly declared in the code. However, constraints required to activate specific exceptions may not be directly declared in the code. In such a case, implicit constraints have to be inferred from exception handling mechanisms. Given that exceptions can be raised in several situations, finding constraints to generate test data to exercise all possible faulty scenarios can significantly increase the number of paths and constraints, which can cause or aggravate path explosion issues. This paper reports on an investigation that we carried out to gauge the cost (i.e., number of path constraints) of four data generation approaches aimed at covering exception dependent paths.Keywords -Exception handling; symbolic execution; test data generation; software analysisAutomatic test data generation is a notorious complex problem. Symbolic execution and constraint solving have been used as an approach to generate test data that achieve high control-flow coverage [5,9]. During symbolic execution, program elements are represented as functions of symbolic input values [5,9]. Each path is represented by a path constraint, which is a sequence of constraints that should be satisfied so that the underlying path can be traversed. Constraint solvers are thus used to generate concrete input values (i.e., test data) that satisfy each set of constraints.Often, the constraints required to traverse a path are explicitly declared in the code by means of control-flow statements, such as if and while. However, some constraints are not explicitly declared in the code because they do not stem from conventional control-flow statements. Some of these constraints implicitly derive from exception handling mechanisms such as Java's try-catch-finally blocks. For instance, a block that declares a NullPointerException will be executed only when such an exception is thrown. However, in general, there are no constraint indicating in which condition such an exception will be thrown.We classify paths that depend on an exception being thrown as exception-dependent paths (EDPs) [7], as oppose to exception-free paths (EFPs) [14]. According to an analysis performed over a sample of 100 open source projects called SF100 [8] 1 , we discovered that almost one third of the methods have at least one EDP [7]. Although the number of EDPs of a program is high, and exception handling is an important topic in software development [4], the influence of exception mechanisms to unit test data generation using symbolic execution has not been widely explored [1,3,5].Taking into account the implicit constraints that stem from exception handling mechanisms can significantly increase the number of path constraints. This has the potential to exacerbate a well-known issue faced by symbolic execution approaches: path explosion [1,5,6], which is usually caused by complex loop structures.The c...

show abstract

“…In the past years, a large number of test case generation techniques have been proposed and investigated; see (Anand et al 2013) for a recent survey. Among the most well known are codebased test generation techniques.…”

mentioning

confidence: 99%

Toward a mature industrial practice of software test automation

et al. 2014

Self Cite

View full text Add to dashboard Cite

Since the launch of the first IEEE/ACM International Workshop on Automation of Software Test eight years ago, the research in the area has developed rapidly as test automation has become a mature practice widely adopted by the IT and software industry.One of the goals that has been continuously pursued is to bridge the gap between research and practice. Many of the recent papers are closely related to the current practice in the industry reflecting the trend toward a mature community unifying both researchers and practitioners. This is also reflected in the diversity of the topics related to the practical industrial problems covered in this special section.Authorization systems are at the heart of computer security. Antonia Bertolino et al.'s paper entitled Testing of PolPA-based Usage Control Systems is devoted to the testing of the Policy Decision Point (PDP) within the authorization system called PolPA, which enables both history-based and usage-based control of accesses. They propose two testing strategies for validating the history-based access control and the usage control functionalities, respectively. The former is based on a fault model being able to highlight the problems and vulnerabilities that could occur during the PDP implementation. The latter combines the standard technique for condition coverage with a methodology for simulating the continuous control of the PDP

show abstract

An orchestrated survey of methodologies for automated software test case generation

Cited by 527 publications

References 189 publications

Similarity-Based Search for Model Checking: A Pilot Study with Java PathFinder

Similarity-Based Search for Model Checking: A Pilot Study with Java PathFinder

Analyzing Exceptions in the Context of Test Data Generation Based on Symbolic Execution

Toward a mature industrial practice of software test automation

Contact Info

Product

Resources

About