Aspect-Based API Review Classification: How Far Can Pre-Trained Transformer Model Go?

Xu, Bowen; Khan, Junaed Younus; Uddin, Gias; Han, DongGyun; Yang, Zhou; Lo, David

doi:10.1109/saner53432.2022.00054

Cited by 23 publications

(7 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results in Table 1 are close to the results reported by Wang [28] that evaluate the three models. 5…”

Section: Settings Of Victim Modelsmentioning

confidence: 99%

“…With the emergence of Open-Source Software (OSS) data and advances in Deep Neural Networks (DNN), recent years have witnessed a dramatic rise in applying DNNbased models to critical software engineering tasks [1], including function name prediction [2], code search [3], clone detection [4], API classification [5], StackOverflow post tagging [6], etc. Meanwhile, the security issues of these models have also become a growing concern.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stealthy Backdoor Attack for Code Models

Zhang¹,

Xu²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Code models, such as CodeBERT and CodeT5, offer general-purpose representations of code and play a vital role in supporting downstream automated software engineering tasks. Most recently, code models were revealed to be vulnerable to backdoor attacks. A code model that is backdoor-attacked can behave normally on clean examples but will produce pre-defined malicious outputs on examples injected with triggers that activate the backdoors. Existing backdoor attacks on code models use unstealthy and easy-to-detect triggers. This paper aims to investigate the vulnerability of code models with stealthy backdoor attacks. To this end, we propose AFRAIDOOR (Adversarial F eatur e as Adaptive Backdoor ). AFRAIDOOR achieves stealthiness by leveraging adversarial perturbations to inject adaptive triggers into different inputs. We evaluate AFRAIDOOR on three widely adopted code models (CodeBERT, PLBART and CodeT5) and two downstream tasks (code summarization and method name prediction). We find that around 85% of adaptive triggers in AFRAIDOOR bypass the detection in the defense process. By contrast, only less than 12% of the triggers from previous work bypass the defense. When the defense method is not applied, both AFRAIDOOR and baselines have almost perfect attack success rates. However, once a defense is applied, the success rates of baselines decrease dramatically to 10.47% and 12.06%, while the success rate of AFRAIDOOR are 77.05% and 92.98% on the two tasks. Our finding exposes security weaknesses in code models under stealthy backdoor attacks and shows that the state-of-the-art defense method cannot provide sufficient protection. We call for more research efforts in understanding security threats to code models and developing more effective countermeasures.

show abstract

“…The results in Table 1 are close to the results reported by Wang [28] that evaluate the three models. 5…”

Section: Settings Of Victim Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Stealthy Backdoor Attack for Code Models

Zhang¹,

Xu²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We have described two representatives of encoder-only models, CodeBERT [10] and GraphCodeBERT [15], in Section 2.1. The two models have demonstrated good performance across multiple software engineering tasks, including API review [46], Stack Overflow post analysis [16], etc. There are some other encoder-only pretrained models of code.…”

Section: Pre-trained Models Of Codementioning

confidence: 99%

Compressing Pre-trained Models of Code into 3 MB

Shi¹,

Zhang²,

Xu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Although large pre-trained models of code have delivered significant advancements in various code processing tasks, there is an impediment to the wide and fluent adoption of these powerful models in software developers' daily workflow: these large models consume hundreds of megabytes of memory and run slowly on personal devices, which causes problems in model deployment and greatly degrades the user experience.It motivates us to propose Compressor, a novel approach that can compress the pre-trained models of code into extremely small models with negligible performance sacrifice. Our proposed method formulates the design of tiny models as simplifying the pre-trained model architecture: searching for a significantly smaller model that follows an architectural design similar to the original pre-trained model. Compressor proposes a genetic algorithm (GA)-based strategy to guide the simplification process. Prior studies found that a model with higher computational cost tends to be more powerful. Inspired by this insight, the GA algorithm is designed to maximize a model's Giga floating-point operations (GFLOPs), an indicator of the model computational cost, to satisfy the constraint of the target model size. Then, we use the knowledge distillation technique to train the small model: unlabelled data is fed into the large model and the outputs are used as labels to train the small model. We evaluate Compressor with two state-of-the-art pre-trained models, i.e., CodeBERT and GraphCodeBERT, on two important tasks, i.e., vulnerability prediction and clone detection. We use our method to compress pre-trained models to a size (3 MB), which is 160× smaller than the original size. The results show that compressed CodeBERT and GraphCodeBERT are 4.31× and 4.15× faster than the original model at inference, respectively. More importantly, they maintain 96.15% and 97.74% of the original performance on the vulnerability prediction task. They even maintain higher ratios (99.20% and 97.52%) of the original performance on the clone detection task. CCS CONCEPTS• Software and its engineering → Search-based software engineering; Designing software; • Computing methodologies → Artificial intelligence.

show abstract

“…Machine learning (ML) projects are becoming increasingly popular and play essential roles in various domain, e.g., code processing [7], [8], self-driving cars, speech recognition [9], etc. Despite widespread usage and popularity, only a few research works try to examine AI and ML projects to identify unique properties, development patterns, and trends.…”

Section: Introductionmentioning

confidence: 99%

“…the code is split into different modules and no ad-hoc scripts; (3) check whether good documentations are provided; (4) check if the project uses issues to track new features and bugs; (5) check if the project uses a CI service, e.g. Travis, CircleCI, etc; (6) check if the project is updated within the last one month;(7) check how many active contributors the project has; (8) check whether the project provides a license.For every point in the guideline, we consider the following dimensions for the project assessment: unit testing for point(1), architecture for point(2), documentation for point (3), issues for point (4), CI for point(5), history for point(6), community for point(7), and license for point(8). Aside from providing a label on whether a project is engineered or not, the labellers also provide descriptive information for every dimension.…”

mentioning

confidence: 99%

NICHE: A Curated Dataset of Engineered Machine Learning Projects in Python

Widyasari,

YANG,

Thung

et al. 2023

View full text Add to dashboard Cite

Machine learning (ML) has gained much attention and been incorporated into our daily lives. While there are numerous publicly available ML projects on open source platforms such as GitHub, there have been limited attempts in filtering those projects to curate ML projects of high quality. The limited availability of such high-quality dataset poses an obstacle in understanding ML projects. To help clear this obstacle, we present NICHE, a manually labelled dataset consisting of 572 ML projects. Based on evidences of good software engineering practices, we label 441 of these projects as engineered and 131 as non-engineered. This dataset can help researchers understand the practices that are followed in high-quality ML projects. It can also be used as a benchmark for classifiers designed to identify engineered ML projects.

show abstract

Aspect-Based API Review Classification: How Far Can Pre-Trained Transformer Model Go?

Cited by 23 publications

References 30 publications

Stealthy Backdoor Attack for Code Models

Stealthy Backdoor Attack for Code Models

Compressing Pre-trained Models of Code into 3 MB

NICHE: A Curated Dataset of Engineered Machine Learning Projects in Python

Contact Info

Product

Resources

About