Towards a unified fault-detection benchmark

Schmeelk, Suzanna

doi:10.1145/1806672.1806684

Cited by 2 publications

(3 citation statements)

References 36 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, Schmeelk [20] has recently introduced the design of a repository, in order to integrate benchmarks with publicly available fault taxonomies like the CWE. He also pinpoints the need for a unified benchmarking framework.…”

Section: Related Workmentioning

confidence: 99%

Test-Driving Static Analysis Tools in Search of C Code Vulnerabilities

Chatzieleftheriou

Katsaros

2011

2011 IEEE 35th Annual Computer Software and Applications Conference Workshops

View full text Add to dashboard Cite

Abstract-Recently, a number of tools for automated code scanning came in the limelight. Due to the significant costs associated with incorporating such a tool in the software lifecycle, it is important to know what defects are detected and how accurate and efficient the analysis is. We focus specifically on popular static analysis tools for C code defects. Existing benchmarks include the actual defects in open source programs, but they lack systematic coverage of possible code defects and the coding complexities in which they arise. We introduce a test suite implementing the discussed requirements for frequent defects selected from public catalogues. Four open source and two commercial tools are compared in terms of their effectiveness and efficiency of their detection capability. A wide range of C constructs is taken into account and appropriate metrics are computed, which show how the tools balance inherent analysis tradeoffs and efficiency. The results are useful for identifying the appropriate tool, in terms of costeffectiveness, while the proposed methodology and test suite may be reused.

show abstract

Section: Related Workmentioning

confidence: 99%

Test-Driving Static Analysis Tools in Search of C Code Vulnerabilities

Chatzieleftheriou

Katsaros

2011

2011 IEEE 35th Annual Computer Software and Applications Conference Workshops

View full text Add to dashboard Cite

show abstract

“…Benchmarks are very important for evaluating pRogram Analysis and Testing (RAT) algorithms and tools [6][7][8][9][10]. Different benchmarks exist to evaluate different RAT aspects, such as how scalable RAT tools are, how fast they can achieve high test coverage, how thorough they handle different 408 I. HUSSAIN ET AL.…”

Section: Introductionmentioning

confidence: 99%

“…Benchmarks are very important for evaluating pRogram Analysis and Testing (RAT) algorithms and tools . Different benchmarks exist to evaluate different RAT aspects, such as how scalable RAT tools are, how fast they can achieve high test coverage, how thorough they handle different language extensions, how well they translate and refactor code, how effective RAT tools are in executing applications symbolically or concolically, and how efficient these tools are in optimizing, linking, and loading code in compiler‐related technologies, as well as profiling.…”

Section: Introductionmentioning

confidence: 99%

RUGRAT: Evaluating program analysis and testing tools and compilers with large generated random benchmark applications

Hussain

Csallner

Grechanik

et al. 2014

Softw. Pract. Exper.

View full text Add to dashboard Cite

Summary Benchmarks are heavily used in different areas of computer science to evaluate algorithms and tools. In program analysis and testing, open‐source and commercial programs are routinely used as benchmarks to evaluate different aspects of algorithms and tools. Unfortunately, many of these programs are written by programmers who introduce different biases, not to mention that it is very difficult to find programs that can serve as benchmarks with high reproducibility of results. We propose a novel approach for generating random benchmarks for evaluating program analysis and testing tools and compilers. Our approach uses stochastic parse trees, where language grammar production rules are assigned probabilities that specify the frequencies with which instantiations of these rules will appear in the generated programs. We implemented our tool for Java and applied it to generate a set of large benchmark programs of up to 5M lines of code each with which we evaluated different program analysis and testing tools and compilers. The generated benchmarks let us independently rediscover several issues in the evaluated tools. Copyright © 2014 John Wiley & Sons, Ltd.

show abstract

Towards a unified fault-detection benchmark

Cited by 2 publications

References 36 publications

Test-Driving Static Analysis Tools in Search of C Code Vulnerabilities

Test-Driving Static Analysis Tools in Search of C Code Vulnerabilities

RUGRAT: Evaluating program analysis and testing tools and compilers with large generated random benchmark applications

Contact Info

Product

Resources

About