A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software

Abstract

Advancing our understanding of software vulnerabilities, automating their identification, the analysis of their impact, and ultimately their mitigation is necessary to enable the development of software that is more secure. While operating a vulnerability assessment tool that we developed and that is currently used by hundreds of development units at SAP, we manually collected and curated a dataset of vulnerabilities of open-source software and the commits fixing them. The data was obtained both from the National Vulnerability Database (NVD) and from project-specific Web resources that we monitor on a continuous basis. From that data, we extracted a dataset that maps 624 publicly disclosed vulnerabilities affecting 205 distinct open-source Java projects, used in SAP products or internal tools, onto the 1282 commits that fix them. Out of 624 vulnerabilities, 29 do not have a CVE identifier at all and 46, which do have a CVE identifier assigned by a numbering authority, are not available in the NVD yet. The dataset is released under an open-source license, together with supporting scripts that allow researchers to automatically retrieve the actual content of the commits from the corresponding repositories and to augment the attributes available for each instance. Also, these scripts allow to complement the dataset with additional instances that are not security fixes (which is useful, for example, in machine learning applications). Our dataset has been successfully used to train classifiers that could automatically identify security-relevant commits in code repositories. The release of this dataset and the supporting code as open-source will allow future research to be based on data of industrial relevance; also, it represents a concrete step towards making the maintenance of this dataset a shared effort involving open-source communities, academia, and the industry.

Automated vulnerability detection system based on commit messages

“…We will compare our results with these two approaches in the evaluation section. Both our approach and [5] performed extensive data gathering to hand-curate the dataset. However our dataset covers much more VFC and wider range of programming languages whereas [5] only considered Java.…”

“…Both our approach and [5] performed extensive data gathering to hand-curate the dataset. However our dataset covers much more VFC and wider range of programming languages whereas [5] only considered Java. The dataset was also heavily focused on industry relevant projects which is not a good indicator of the whole opensource scene in the real world.…”

“…However, these tools perform only shallow understanding of the code and may produce many false positives. For example, the author of FlawFinder [5] states that the tool primarily uses simple text pattern matching and does not understand the semantics of the code at all and furthermore advises to use the tool only as guide in finding and removing security vulnerabilities.…”

“…The majority of the static analysis tools have been focused on analyzing software written in high-level languages, e.g. FlawFinder [5], Splint [6], Microsoft PREfast [7] for C/C++, FindBugs [8] for Java and PMD [9] for multiple languages (Java, JavaScript, XML).…”

“…The dataset is also published by the authors for free to the public. Based on the dataset, they've trained a Support Vector Machine (SVM) classifier to flag suspicious commis which outperformed FlawFinder[5] by reducing false positives by 99% while maintaining the same recall rate. This is very close to our work, but looking at the problem at a slightly different angel, as our work mainly focus on finding VFC instead of VCC.…”