A Practical Approach to the Automatic Classification of Security-Relevant Commits

Abstract

The lack of reliable sources of detailed information on the vulnerabilities of open-source software (OSS) components is a major obstacle to maintaining a secure software supply chain and an effective vulnerability management process. Standard sources of advisories and vulnerability data, such as the National Vulnerability Database (NVD), are known to suffer from poor coverage and inconsistent quality. To reduce our dependency on these sources, we propose an approach that uses machine-learning to analyze source code repositories and to automatically identify commits that are security-relevant (i.e., that are likely to fix a vulnerability). We treat the source code changes introduced by commits as documents written in natural language, classifying them using standard document classification methods. Combining independent classifiers that use information from different facets of commits, our method can yield high precision (80%) while ensuring acceptable recall (43%). In particular, the use of information extracted from the source code changes yields a substantial improvement over the best known approach in state of the art, while requiring a significantly smaller amount of training data and employing a simpler architecture.

A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software

“…Another example of a possible application of the dataset is presented in [8]. Motivated by the need to automate the Vulnerability Id.…”

“…Commits CVE-2015-5348 23 CVE-2012-0022 15 CVE-2018-8009 13 CVE-2016-6801 13 CVE-2016-8749 12 CVE-2018-8027 12 CVE-2014-0119 11 CVE-2012-2098 10 CVE-2013- Commits 1 181 2 166 3 256 4 259 5 308 6 282 7 257 8 136 9 144 10 172 11 170 12 129 13 128 14 512 15 maintenance of the very vulnerability database from which our dataset is extracted, Sabetta and Bezzi [8] presented a novel approach to the automated classification of commits that are security-relevant (i.e., that are likely to fix a vulnerability). They used (an older, and smaller version of) the dataset presented here to train two independent classifiers, considering, respectively, the patch introduced by a commit (Patch Classifier) and the log messages (Message Classifier), without relying on information from vulnerability advisories.…”

Software Vulnerability Classification Based On Deep Neural Network

“…al. [7], a method that uses machine-learning to investigate ASCII text file repositories and to mechanically determine commits that area unit security-relevant (i.e., that area unit probably to mend vulnerability). They treat the ASCII text file changes introduced by commits as documents written in language, classifying them victimization commonplace document classification strategies.…”

Combining Graph-Based Learning With Automated Data Collection for Code Vulnerability Detection

“…Prior work on vulnerable training sample collection [18,19,20] is characterized by a one-size-fits-all assumption. They use a single monolithic model for locating vulnerability-relevant commits.…”

“…We thoroughly evaluate FUNDED on large real-life datasets of code commit history and vulnerable programs written in C, Java, Php and Swift. We compare FUNDED against six state-of-the-art (SOTA) learning-based detection methods for software bugs or vulnerabilities [4,5,16,6,3,15], and five SOTA methods for automatic vulnerable code sample collection [18,19,23,20,24]. Experimental results show that FUNDED consistently outperforms competing methods across evaluation settings, by discovering more code vulnerabilities with a lower false-positive rate.…”