Noisy Label Learning for Security Defects

Abstract

Data-driven software engineering processes, such as vulnerability prediction heavily rely on the quality of the data used. In this paper, we observe that it is infeasible to obtain a noise-free security defect dataset in practice. Despite the vulnerable class, the non-vulnerable modules are difficult to be verified and determined as truly exploit free given the limited manual efforts available. It results in uncertainty, introduces labeling noise in the datasets and affects conclusion validity. To address this issue, we propose novel learning methods that are robust to label impurities and can leverage the most from limited label data; noisy label learning. We investigate various noisy label learning methods applied to software vulnerability prediction. Specifically, we propose a two-stage learning method based on noise cleaning to identify and remediate the noisy samples, which improves AUC and recall of baselines by up to 8.9% and 23.4%, respectively. Moreover, we discuss several hurdles in terms of achieving a performance upper bound with semi-omniscient knowledge of the label noise. Overall, the experimental results show that learning from noisy labels can be effective for data-driven software and security analytics.

Data Quality for Software Vulnerability Datasets

“…In reality, complete knowledge of the latent vulnerabilities is unobtainable. Croft et al [13] observed at least twice as many latent vulnerabilities as known vulnerabilities in their dataset.…”

“…CodeBERT is a pre-trained state-of-the-art code embedding model based on the RoBERTa architecture [52]. Similar studies have demonstrated the effectiveness of Code-BERT for SVP [8], [13]. LineVul generates function-level predictions using a transformer-based architecture.…”

“…Hence, data quality issues will hinder the reliability and trustworthiness of the outcomes. For instance, previous studies [13], [33] have highlighted inflated performance due to inaccurate labelling mechanisms for nonvulnerable modules. Consequently, the industry value and adoption of SVP models is uncertain [37], [38].…”

“…This largely relates to the semantic label correctness; i.e., whether or not data points labelled as vulnerable or non-vulnerable genuinely align. It has previously been observed that non-vulnerable labels are unreliable in real-world datasets as there is no ground truth label source for this class [10], [13], [33]. No oracle can reliably ensure the security and absence of exploits in a given code snippet.…”

“…Nonetheless, software vulnerability data collection is not a trivial task [10]. Labelled examples of software vulnerabilities are difficult to obtain in the real-world, as they are scarce [11], poorly documented [12], and limited to reported vulnerabilities [13]. Consequently, many researchers have conducted labourious work constructing large-scale software vulnerability datasets [14]- [17].…”