APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

Abstract

Security vulnerability fixes could be a promising research avenue for Automated Program Repair (APR) techniques. In recent years, APR tools have been thoroughly developed for fixing generic bugs. However, the area is still relatively unexplored when it comes to fixing security bugs or vulnerabilities. In this paper, we evaluate nine state-of-the-art APR tools and one vulnerability-specific repair tool. In particular, we investigate their ability to generate patches for 79 real-world Java vulnerabilities in the Vul4J dataset, as well as the level of trustworthiness of these patches. We evaluate the tools with respect to their ability to generate security patches that are (i) testable, (ii) having the positive effect of closing the vulnerability, and (iii) not having side effects from a functional point of view. Our results show that the evaluated APR tools were able to generate testable patches for around 20% of the considered vulnerabilities. On average, nearly 73% of the testable patches indeed eliminate the vulnerabilities, but only 44% of them could actually fix security bugs while maintaining the functionalities. To understand the root cause of this phenomenon, we conduct a detailed comparative study of the general bug fix patterns in Defect4J and the vulnerability fix patterns in ExtraVul (which we extend from Vul4J). Our investigation shows that, although security patches are short in terms of lines of code, they contain unique characteristics in their fix patterns compared to general bugs. For example, many security fixes require adding method calls. These method calls contain specific input validation-related keywords, such as encode, normalize, and trim. In this regard, our study suggests that additional repair patterns should be implemented for existing APR tools to fix more types of security vulnerabilities.

An Incremental Lightweight Method for Vulnerability Data Collection for Security Patches

“…Their vulnerability dataset is from CVE [19], Smart Contract Weakness Classification (SWC) [20], and Smart Bugs curated as a ground truth from evaluations. The empirical study [5] has shown the ability of APR tools to generate the trustworthiness of patches for Java security vulnerabilities. The study [3] employs the abstract syntax tree (AST) path technique for representation, enabling the capture of structural information from AST nodes.…”

“…The process aims to reduce the manual effort required in debugging and can be particularly useful in large codebases. Our dataset can be utilized in a similar manner as implemented by the existing studies in automated patch generation [31,32], security vulnerability repairing [33,34], and program repair [5,35]. In addition, in the field of research in software engineering, techniques and tools are developed to fix software vulnerabilities automatically.…”

“…Research in APR has been extensively explored for programming language-specific fixes, such as C/C++ [4], Java [5][6][7], PHP [8], and JavaScript [9], and so on. However, there is little research on source code patch identification and prediction due to the lack of aggregated benchmark datasets for multiple programming languages.…”