An Empirical Study of Rule-Based and Learning-Based Approaches for Static Application Security Testing

Abstract

Background: Static Application Security Testing (SAST) tools purport to assist developers in detecting security issues in source code. These tools typically use rule-based approaches to scan source code for security vulnerabilities. However, due to the significant shortcomings of these tools (i.e., high false positive rates), learning-based approaches for Software Vulnerability Prediction (SVP) are becoming a popular approach. Aims: Despite the similar objectives of these two approaches, their comparative value is unexplored. We provide an empirical analysis of SAST tools and SVP models, to identify their relative capabilities for source code security analysis. Method: We evaluate the detection and assessment performance of several common SAST tools and SVP models on a variety of vulnerability datasets. We further assess the viability and potential benefits of combining the two approaches. Results: SAST tools and SVP models provide similar detection capabilities, but SVP models exhibit better overall performance for both detection and assessment. Unification of the two approaches is difficult due to lacking synergies. Conclusions: Our study generates 12 main findings which provide insights into the capabilities and synergy of these two approaches. Through these observations we provide recommendations for use and improvement. CCS CONCEPTS• Security and privacy → Software security engineering; • Computing methodologies → Machine learning; • Software and its engineering → Software testing and debugging.

An Empirical Study of Developers' Discussions about Security Challenges of Different Programming Languages

“…Language consideration could also be potentially strengthened in several research areas. For instance, vulnerability prediction models often do not consider the characteristics or features of the languages that they target (Yang et al 2017;Croft et al 2021).…”

Incivility Detection in Open Source Code Review and Issue Discussions

“…Finally, we computed the Matthews Correlation Coefficient (MCC) [70], which is a single-value classification metric that is more interpretable and robust to changes in the prediction goal [71] because it summarizes the results of all four quadrants of a confusion matrix, i.e., true positive, false negative, true negative, and false positive [72]. The MCC metric is calculated as MCC =…”

Comparing Effectiveness and Efficiency of Interactive Application Security Testing (Iast) and Runtime Application Self-Protection (Rasp) Tools in A Large Java-Based System

“…Other studies, such as [20], focus on SAST tools for detecting vulnerabilities and find that the true positive and false positive rates vary widely across tools and across different types of vulnerabilities. Croft et al [21] compares open-source, rule-based SAST tools with learning-based software vulnerability prediction models for C/C++ software systems. Croft et al [21] concluded that although learning-based approaches had better precision, both learning-based and SAST tools approaches should be used independently.…”