Revisiting the VCCFinder approach for the identification of vulnerability-contributing commits

Abstract

Detecting vulnerabilities in software is a constant race between development teams and potential attackers. While many static and dynamic approaches have focused on regularly analyzing the software in its entirety, a recent research direction has focused on the analysis of changes that are applied to the code. VCCFinder is a seminal approach in the literature that builds on machine learning to automatically detect whether an incoming commit will introduce some vulnerabilities. Given the influence of VCCFinder in the literature, we undertake an investigation into its performance as a state-of-the-art system. To that end, we propose to attempt a replication study on the VCCFinder supervised learning approach. The insights of our failure to replicate the results reported in the original publication informed the design of a new approach to identify vulnerability-contributing commits based on a semi-supervised learning technique with an alternate feature set. We provide all artefacts and a clear description of this approach as a new reproducible baseline for advancing research on machine learning-based identification of vulnerability-introducing commits.

A Comparative Study of Commit Representations for JIT Vulnerability Prediction

“…Kamei et al uses source-code metrics and Kim et al uses features extracted from the revision history of software projects [13,14]. Similarly, other works extract features from metrics and/or commit metadata to enhance the procedure of code quality assessment and ultimately prevent vulnerabilities [9,15]. Usually, in these methods, the extracted features are fed into a machine learning model, such as an SVM or a random forest.…”

Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review

“…Although some studies attempted to remediate this by making their datasets available, not many of them made the code and methods used to create these datasets available, which has made reproduction and adaptation efforts very difficult. Riom et al [75] found the replication of a seminal work [P16] infeasible due to these challenges.…”