Report on the Static Analysis Tool Exposition (SATE) IV

Abstract

The NIST Software Assurance Metrics And Tool Evaluation (SAMATE) project conducted the fourth Static Analysis Tool Exposition (SATE IV) to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets, encourage improvements to tools, and promote broader and more rapid adoption of tools by objectively demonstrating their use on production software. Briefly, eight participating tool makers ran their tools on a set of programs. The programs were four pairs of large code bases selected in regard to entries in the Common Vulnerabilities and Exposures (CVE) dataset and approximately 60 000 synthetic test cases, the Juliet 1.0 test suite. NIST researchers analyzed approximately 700 warnings by hand, matched tool warnings to the relevant CVE entries, and analyzed over 180 000 warnings for Juliet test cases by automated means. The results and experiences were reported at the SATE IV Workshop in McLean, VA, in March, 2012. The tool reports and analysis were made publicly available in January, 2013. SATE is an ongoing research effort with much work still to do. This paper reports our analysis to date which includes much data about weaknesses that occur in software and about tool capabilities. Our analysis is not intended to be used for tool rating or tool selection. This paper also describes the SATE procedure and provides our observations based on the data collected. Based on lessons learned from our experience with previous SATEs, we made the following major changes to the SATE procedure. First, we introduced the Juliet test suite that has precisely characterized weaknesses. Second, we improved the procedure for characterizing vulnerability locations in the CVE-selected test cases. Finally, we provided teams with a virtual machine image containing the test cases properly configured to compile the cases and ready for analysis by tools. This paper identifies several ways in which the released data and analysis are useful. First, the output from running many tools on production software is available for empirical research. Second, our analysis of tool reports indicates the kinds of weaknesses that exist in the software and that are reported by the tools. Third, the CVE-selected test cases contain exploitable vulnerabilities found in practice, with clearly identified locations in the code. These test cases can help practitioners and researchers improve existing tools and devise new techniques. Fourth, tool outputs for Juliet cases provide a rich set of data amenable to mechanical analysis. Finally, the analysis may be used as a basis for a further study of weaknesses in code and of static analysis.

On the capability of static code analysis to detect security vulnerabilities

“…For our experiment, we used over 21,000 test cases for 22 C/C++ CWEs and over 7500 test cases for 19 Java CWEs. Previous evaluations based on Juliet either did not report detailed quantitative results [13,14] or used a very small sample of only 152 test cases related to vulnerabilities in C code only [15]. Our study reports several performance metrics (i.e., accuracy, recall, probability of false alarm, and G-score) for individual CWEs, as well as across all considered CWEs.…”

“…Note that the exact matching is likely to underestimate the tools performance because CWEs are related and form a hierarchical structure in which lower level CWEs are more specific instances of their parent(s) CWEs. Earlier works used groupings of CWEs in the evaluation [14], but concluded that these groupings have to be improved. In general, due to the complex relationships among CWEs, there is no easy and consistent way to group CWEs in groups of related types of weaknesses [14].…”

“…We start with description of related works that used the Juliet benchmark as an input in evaluation of static code analysis tools [13][14][15][16][17]. These works are the closest to our work.…”

“…Consequently, warnings labeled CWE 129, CWE 20, and CWE 633 may refer to the same weakness, that is, if a static code analysis tool reports CWE 20 or CWE 633 for a bad test case related to CWE 129, it is justifiable to treat that as a 'match' and count the outcome as TP. Note that in the NIST's report on the Static Analysis Tool Exposition (SATE) IV [14] CWEs were grouped into categories that were then used for evaluation, but concluded that improvements of the groupings have to be made to improve the evaluation accuracy. Instead of coming up with groups of CWEs, we decided to rely on the existing relationships in the CWE taxonomy.…”

“…NIST's report presented at SATE IV [14] provided details on evaluating five tools on Juliet 1.0 test suite. The reported results included the number of true positives and false positives for thirteen weakness categories (i.e., groups of CWEs) and it was concluded that the analysis can be improved by changing the categories and fine-tuning the procedure for matching the warnings to test cases.…”