Automated Software Test Generation: Some Challenges, Solutions, and Recent Advances

Candea, George; Godefroid, Patrice

doi:10.1007/978-3-319-91908-9_24

Cited by 21 publications

(11 citation statements)

References 101 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DenseNet's developers discovered the bug and fixed it (by adding an int conversion at line 12) in a later project revision. 4 DenseNet's implementation is deceivingly simple: despite its small size and linear structure, it only accepts input arguments in very specific ranges. Argument input _ shape, for example, corresponds to a so-called shape tuple of integers; in DenseNet, it should be a triple of integers with first element at least 20.…”

Section: An Example Of Using Annotestmentioning

confidence: 99%

An Annotation-based Approach for Finding Bugs in Neural Network Programs

Rezaalipour¹,

Furia²

2021

Preprint

View full text Add to dashboard Cite

As neural networks are increasingly included as core components of safety-critical systems, developing effective testing techniques specialized for them becomes crucial. The bulk of the research has focused on testing neural-network models (for instance, their robustness and reliability as classifiers). But neural-network models are defined by writing programs (usually written in a programming language like Python), and there is growing evidence that these neural-network programs often have bugs. Thus, being able to effectively test neural-network programs is instrumental to their dependability.This paper presents aNNoTest: an approach to generating test inputs for neural-network programs. A fundamental challenge is that the dynamically-typed languages (e.g., Python) used to program neural networks cannot express detailed constraints about valid function inputs (e.g., vectors and matrices with certain dimensions). Without knowing these constraints, automated test-case generation is prone to producing many invalid inputs, which trigger spurious failures and are useless for identifying real bugs. To address this problem, we introduce a simple annotation language tailored for expressing valid function inputs in neural-network programs. aNNo-Test inputs an annotated program, and uses property-based testing to generate random inputs that satisfy the validity constraints. In the paper, we also outline guidelines that help reduce the effort needed to write aNNoTest annotations.We evaluated aNNoTest on 19 neural-network programs from Islam et al. 's survey [13], which we manually annotated following our guidelines-producing 6 annotations per tested function on average. aNNoTest automatically generated test inputs that revealed 94 bugs, including 63 bugs that the survey reported for these projects. These results suggest that aNNoTest can be a cost-effective approach to finding widespread bugs in neural-network programs.

show abstract

Section: An Example Of Using Annotestmentioning

confidence: 99%

An Annotation-based Approach for Finding Bugs in Neural Network Programs

Rezaalipour¹,

Furia²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Also the approaches presented in the remaining papers could profit from DSLs, e.g., as follows: [41] for specifying certain assertions or contracts, [43] for specifying data flow analyses 20 , [25] for specifying test models, [42] for defining learning alphabets or representing the learning result, [40] for modularly specifying the required code instrumentation, e.g. in an aspect-oriented fashion, and [15,19,66] for conveniently specifying their enriched system structures.…”

Section: Volume-related Interrelationsmentioning

confidence: 99%

“…On the other hand, LDE could also profit from the approaches presented in the other papers. In particular, all the involved analysis, verification and validation methods of [15,19,25,[40][41][42][43]61,66] are good candidates for inclusion in mIDEs in order to improve the development support and/or to control nonfunctional constraints. Finally, [27] provides a wealth of observations and techniques with potential to impact the future mIDE development.…”

Section: Volume-related Interrelationsmentioning

confidence: 99%

Language-Driven Engineering: From General-Purpose to Purpose-Specific Languages

Steffen

Gossen

Naujokat

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In this paper, we present the paradigm of Language-Driven Engineering (LDE), which is characterized by its unique support for division of labour on the basis of Domain-Specific Languages (DSLs) targeting different stakeholders. LDE allows the involved stakeholders, including the application experts, to participate in the system development and evolution process using dedicated DSLs, while at the same time establishing new levels of reuse that are enabled by powerful model transformations and code generation. Technically, the interplay between the involved DSLs is realized in a service-oriented fashion. This eases a product line approach and system evolution by allowing to introduce and exchange entire DSLs within corresponding Mindset-Supporting Integrated Development Environments (mIDEs). The impact of this approach is illustrated along the development and evolution of a profilebased email distribution system. Here we do not want to emphasize the precise choice of DSLs, but rather the flexible DSL-based modularization of the development process, which allows one to freely introduce and exchange DSLs as needed to optimally capture the mindsets of the involved stakeholders.

show abstract

“…Automated Software Test Generation: Some Challenges, Solutions, and Recent Advances [5] discusses automated test generation from a practical perspective. After explaining random testing and input fuzzing, the chapter turns to test generation via dynamic symbolic execution, whose precision improves over 'traditional', for example, coverage-heuristics-based test generation approaches.…”

Section: Validation: Testing and Beyondmentioning

confidence: 99%

Methods, Languages and Tools for Future System Development

Steffen

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Language design for simplifying programming, analysis/verification methods and tools for guaranteeing, for example, security and real-time constraints, and validation environments for increasing automation during quality assurance can all be regarded as means to factor out and generically solve specific concerns of the software development process and then reuse the corresponding solutions. In this sense, reuse, a guiding engineering principle, appears as a unifying theme in software science, and it is not surprising that the corresponding research is continuously converging. The following summary of the contributions of the second topical part of the celebration volume LNCS 10,000 aims at establishing a common perspective and indicating the state and progress of this convergence.

show abstract

Automated Software Test Generation: Some Challenges, Solutions, and Recent Advances

Cited by 21 publications

References 101 publications

An Annotation-based Approach for Finding Bugs in Neural Network Programs

An Annotation-based Approach for Finding Bugs in Neural Network Programs

Language-Driven Engineering: From General-Purpose to Purpose-Specific Languages

Methods, Languages and Tools for Future System Development

Contact Info

Product

Resources

About