The second static analysis tool exposition (SATE) 2009

Okun, Vadim; Delaitre, Aurélien; Black, Paul E.

doi:10.6028/nist.sp.500-287

Cited by 14 publications

(11 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For SATE 2009, SATE 2010, and SATE IV, we used a similar approach [7][8][9]. Security experts performed time-limited analyses of some of the test cases to identify the most important weaknesses.…”

Section: Related Workmentioning

confidence: 99%

“…To address this problem in SATE 2009 [7] and the following SATEs, we randomly selected a subset of thirty warnings from each tool report, based on weakness category and severity. The selection procedure assigned higher weight to higher severity warnings.…”

Section: 8mentioning

confidence: 99%

“…In 2008, we initiated the first large-scale public event, inviting toolmakers to demonstrate the use of their tools. We labeled it the Static Analysis Tool Exposition (SATE) and refined it over five instances [6][7][8][9]. SATE is designed to advance research in static analysis tools that find security-relevant weaknesses in source code.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SATE V report: ten years of static analysis tool expositions

Delaitre¹,

Stivalet²,

Black³

et al. 2018

Self Cite

View full text Add to dashboard Cite

Software assurance has been the focus of the National Institute of Standards and Technology (NIST) Software Assurance Metrics and Tool Evaluation (SAMATE) team for many years. The Static Analysis Tool Exposition (SATE) is one of the team's prominent projects to advance research in and adoption of static analysis, one of several software assurance methods. This report describes our approach and methodology. It then presents and discusses the results collected from the fifth edition of SATE. Overall, the goal of SATE was not to rank static analysis tools, but rather to propose a methodology to assess tool effectiveness. Others can use this methodology to determine which tools fit their requirements. The results in this report are presented as examples and used as a basis for further discussion. Our methodology relies on metrics, such as recall and precision, to determine tool effectiveness. To calculate these metrics, we designed test cases that exhibit certain characteristics. Most of the test cases were large pieces of software with cybersecurity implications. Fourteen participants ran their tools on these test cases and sent us a report of their findings. We analyzed these reports and calculated the metrics to assess the tools' effectiveness. Although a few results remained inconclusive, many key elements could be inferred based on our methodology, test cases, and analysis. In particular, we were able to estimate the propensity of tools to find critical vulnerabilities in real software, the degree of noise they produced, and the type of weaknesses they were able to find. Some shortcomings in the methodology and test cases were also identified and solutions proposed for the next edition of SATE.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: 8mentioning

confidence: 99%

See 1 more Smart Citation

SATE V report: ten years of static analysis tool expositions

Delaitre¹,

Stivalet²,

Black³

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…The data also provided evidence that, while human analysis is best suited for identifying some types of weaknesses, for instance, improper access control, tools find a significant portion of weaknesses considered important by experts [26].…”

Section: Static Analysis Tool Exposition (Sate)mentioning

confidence: 97%

“…SATE 2009 showed that "tools find weaknesses in many important weakness categories and can quickly identify and describe in detail many weakness instances." [26] Like spell checkers, these tools cannot catch everything. A spell checker will not turn your impromptu tweet into Shakespeare, but spell checkers are helpful enough that they are ubiquitous.…”

Section: Static Analysis Tool Exposition (Sate)mentioning

confidence: 99%

Counting Bugs is Harder Than You Think

Black

2011

2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation

View full text Add to dashboard Cite

Software Assurance Metrics And Tool Evaluation (SAMATE) is a broad, inclusive project at the U.S. National Institute of Standards and Technology (NIST) with the goal of improving software assurance by developing materials, specifications, and methods to test tools and techniques and measure their effectiveness. We review some SAMATE sub-projects: web application security scanners, malware research protocol, electronic voting systems, the SAMATE Reference Dataset, a public repository of thousands of example programs with known weaknesses, and the Static Analysis Tool Exposition (SATE). Along the way we list over two dozen possible research questions, which are also collaboration opportunities.Software metrics are incomplete without metrics of what is variously called bugs, flaws, or faults. We detail numerous critical research problems related to such metrics. For instance, is a warning from a source code scanner a real bug, a false positive, or something else? If a numeric overflow leads to buffer overflow, which leads to command injection, what is the error? How many bugs are there if two sources call two sinks: 1, 2, or 4? Where is a missing feature? We conclude with a list of concepts which may be a useful basis of bug metrics.

show abstract

Static Program Analysis

Shrestha¹

2011

Encyclopedia of Cryptography and Security

View full text Add to dashboard Cite

The second static analysis tool exposition (SATE) 2009

Cited by 14 publications

References 4 publications

SATE V report: ten years of static analysis tool expositions

SATE V report: ten years of static analysis tool expositions

Counting Bugs is Harder Than You Think

Static Program Analysis

Contact Info

Product

Resources

About