The NIST Software Assurance Metrics And Tool Evaluation (SAMATE) project conducted the fourth Static Analysis Tool Exposition (SATE IV) to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets, encourage improvements to tools, and promote broader and more rapid adoption of tools by objectively demonstrating their use on production software. Briefly, eight participating tool makers ran their tools on a set of programs. The programs were four pairs of large code bases selected in regard to entries in the Common Vulnerabilities and Exposures (CVE) dataset and approximately 60 000 synthetic test cases, the Juliet 1.0 test suite. NIST researchers analyzed approximately 700 warnings by hand, matched tool warnings to the relevant CVE entries, and analyzed over 180 000 warnings for Juliet test cases by automated means. The results and experiences were reported at the SATE IV Workshop in McLean, VA, in March, 2012. The tool reports and analysis were made publicly available in January, 2013. SATE is an ongoing research effort with much work still to do. This paper reports our analysis to date which includes much data about weaknesses that occur in software and about tool capabilities. Our analysis is not intended to be used for tool rating or tool selection. This paper also describes the SATE procedure and provides our observations based on the data collected. Based on lessons learned from our experience with previous SATEs, we made the following major changes to the SATE procedure. First, we introduced the Juliet test suite that has precisely characterized weaknesses. Second, we improved the procedure for characterizing vulnerability locations in the CVE-selected test cases. Finally, we provided teams with a virtual machine image containing the test cases properly configured to compile the cases and ready for analysis by tools. This paper identifies several ways in which the released data and analysis are useful. First, the output from running many tools on production software is available for empirical research. Second, our analysis of tool reports indicates the kinds of weaknesses that exist in the software and that are reported by the tools. Third, the CVE-selected test cases contain exploitable vulnerabilities found in practice, with clearly identified locations in the code. These test cases can help practitioners and researchers improve existing tools and devise new techniques. Fourth, tool outputs for Juliet cases provide a rich set of data amenable to mechanical analysis. Finally, the analysis may be used as a basis for a further study of weaknesses in code and of static analysis.

Abstract:The NIST Software Assurance Metrics And Tool Evaluation (SAMATE) project conducted the third Static Analysis Tool Exposition (SATE) in 2010 to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets, encourage improvements to tools, and promote broader and more rapid adoption of tools by objectively demonstrating their use on production software.Briefly, participating tool makers ran their tool on a set of programs. Researchers led by NIST performed a partial analysis of tool reports. The results and experiences were reported at the SATE 2010 Workshop in Gaithersburg, MD, in October, 2010. The tool reports and analysis were made publicly available in 2011.This special publication consists of the following three papers. "The Third Static Analysis Tool Exposition (SATE 2010)," by Vadim Okun, Aurelien Delaitre, and Paul E. Black, describes the SATE procedure and provides observations based on the data collected. The other two papers are written by participating tool makers."Goanna Static Analysis at the NIST Static Analysis Tool Exposition," by Mark Bradley, Ansgar Fehnker, Ralf Huuck, and Paul Steckler, introduces Goanna, which uses a combination of static analysis with model checking, and describes its SATE experience, tool results, and some of the lessons learned in the process. Serguei A. Mokhov introduces a machine learning approach to static analysis and presents MARFCAT's SATE 2010 results in "The use of machine learning with signaland NLP processing of source code to fingerprint, detect, and classify vulnerabilities and weaknesses with MARFCAT." Keywords:Software security; static analysis tools; security weaknesses; vulnerability Certain instruments, software, materials, and organizations are identified in this paper to specify the exposition adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the instruments, software, or materials are necessarily the best available for the purpose.NIST SP 500-283 2 This paper describes the SATE procedure and provides our observations based on the data collected. We improved the procedure based on lessons learned from our experience with previous SATEs. One improvement was selecting programs based on entries in the Common Vulnerabilities and Exposures (CVE) dataset. Other improvements were selection of tool warnings that identify the CVE entries, expanding the C track to a C/C++ track, having largerup to 4 million lines of code -test cases, clarifying further the analysis categories, and having much more detailed analysis criteria.This paper identifies several ways in which the released data and analysis are useful. First, the output from running many tools on production software can be used for empirical research. Second, the analysis of tool reports indicates actual weaknesses that exist in the software and that are reported by the tools.Third, the CVE-selected test cases conta...

Abstract:Cell

Software assurance has been the focus of the National Institute of Standards and Technology (NIST) Software Assurance Metrics and Tool Evaluation (SAMATE) team for many years. The Static Analysis Tool Exposition (SATE) is one of the team's prominent projects to advance research in and adoption of static analysis, one of several software assurance methods. This report describes our approach and methodology. It then presents and discusses the results collected from the fifth edition of SATE. Overall, the goal of SATE was not to rank static analysis tools, but rather to propose a methodology to assess tool effectiveness. Others can use this methodology to determine which tools fit their requirements. The results in this report are presented as examples and used as a basis for further discussion. Our methodology relies on metrics, such as recall and precision, to determine tool effectiveness. To calculate these metrics, we designed test cases that exhibit certain characteristics. Most of the test cases were large pieces of software with cybersecurity implications. Fourteen participants ran their tools on these test cases and sent us a report of their findings. We analyzed these reports and calculated the metrics to assess the tools' effectiveness. Although a few results remained inconclusive, many key elements could be inferred based on our methodology, test cases, and analysis. In particular, we were able to estimate the propensity of tools to find critical vulnerabilities in real software, the degree of noise they produced, and the type of weaknesses they were able to find. Some shortcomings in the methodology and test cases were also identified and solutions proposed for the next edition of SATE.

No abstract