Towards modeling the behavior of static code analysis tools

On the capability of static code analysis to detect security vulnerabilities

“…We start with description of related works that used the Juliet benchmark as an input in evaluation of static code analysis tools [13][14][15][16][17]. These works are the closest to our work.…”

“…Recently, researchers have started to use Juliet test suite for evaluation of static code analysis [15,17]. A small subset of Juliet test suite was used in [15] to compare the performance of nine tools in detecting security vulnerabilities in C code.…”

“…(These values were called 'absolute' in [15] and were obtained by taking into account only the vulnerabilities that specific tools were designed to detect, that is, 'vulnerabilities not covered' were excluded.) A 2014 study [17] evaluated two commercial static analysis tools (which were not identified) using the Juliet test suite. This work did not provide detailed results; it only provided a bar graph of recall values per CWE and the true positive, false positive, and false negative numbers for two selected CWEs (i.e., CWE 134 and CWE 401).…”

Identifying and Documenting False Positive Patterns Generated by Static Code Analysis Tools

“…More specifically, we integrated each static code analysis tool into the Static Code Analysis Tool Evaluator (SCATE) [19], a framework for evaluating the quality of static code analysis tools. SCATE uses rules similar to the foregoing to automatically classify static code analysis tool warnings in the Juliet test suite.…”

“…For example, one study evaluated how different static code analysis tools performed on the Juliet test suite [18] from the National Institute of Standards and Technology and discovered a large number of false positives [19]. For two commercial off-the-shelf static code analysis tools used in the study, as many as 59% and 63%, respectively, of the warnings generated were false positives.…”