Vulnerability Prediction Models: A Case Study on the Linux Kernel

Abstract

Abstract-To assist the vulnerability identification process, researchers proposed prediction models that highlight (for inspection) the most likely to be vulnerable parts of a system. In this paper we aim at making a reliable replication and comparison of the main vulnerability prediction models. Thus, we seek for determining their effectiveness, i.e., their ability to distinguish between vulnerable and non-vulnerable components, in the context of the Linux Kernel, under different scenarios. To achieve the above-mentioned aims, we mined vulnerabilities reported in the National Vulnerability Database and created a large dataset with all vulnerable components of Linux from 2005 to 2016. Based on this, we then built and evaluated the prediction models. We observe that an approach based on the header files included and on function calls performs best when aiming at future vulnerabilities, while text mining is the best technique when aiming at random instances. We also found that models based on code metrics perform poorly. We show that in the context of the Linux kernel, vulnerability prediction models can be superior to random selection and relatively precise. Thus, we conclude that practitioners have a valuable tool for prioritizing their security inspection efforts.

The importance of accounting for real-world labelling when predicting software vulnerabilities

“…Jimenez et al [22] carried out an empirical study comparing the vulnerability prediction approaches using a dataset of 743 vulnerabilities from the Linux Kernel (which was split into independent training and evaluation data sets) and found that function calls and text mining were the best performing approaches. Although related, Jimenez et al used a commit-based analysis for only one system (while herein we use a release-based one for three systems) and does not investigate the impact of data leakage.…”

Static Analysis-Based Approaches for Secure Software Development

“…Although it is a relatively new area of research, a great number of VPMs has already been proposed in the related literature. As stated in [9], the main VPMs that can be found in the literature utilize software metrics [13][14][15][16][17][18][19][20][21][22], text mining [23][24][25][26][27][28], and security-related static analysis alerts [10,[29][30][31][32]] to predict vulnerabilities. These types of VPMs are analyzed in the rest of this section.…”

“…Finally, different empirical studies have shown that text mining-based models exhibit better predictive performance in comparison to other state-of-the-art techniques [9,33,34]. However, they perform poorly in cross-project prediction, which indicates that they are highly projectspecific [33], while excessive amount of time and memory is required for their construction and regular application [9,34].…”

“…In fact, recent studies have highlighted the ability of ASA tools to predict software components that contain vulnerabilities (i.e. vulnerable components) [9,10]. The prediction outcome may provide useful insights to project managers about where to focus their testing efforts.…”

Vulnerability Prediction Based on Weighted Software Network for Secure Software Building

“…Compared with the other vulnerability analysis techniques (e.g., [11,12]), VPM is used as a guidance before security testing. At present, a lot of VPMs have been proposed [6][7][8][9][10][11][12][13][14][15][16][17][18][19]. Software metrics are utilized as features in these VPMs, and the training data are obtained from public vulnerabilities database.…”