AUGER: automatically generating review comments with pre-training models

Li, Lingwei; Yang, Linzhang; Jiang, Huaxi; Yan, Jun; Luo, Tiejian; Hua, Zihan; Li, Geng; Zuo, Chun

doi:10.1145/3540250.3549099

Cited by 17 publications

(1 citation statement)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides the BLEU score, we also choose ROUGE [43] as the metric as it is widely used for evaluating the automatic summarization of texts [37]. We employ ROUGE-1 score, which measures the overlap of words between hypothesis and reference.…”

Section: Evaluation Metricsmentioning

confidence: 99%

On the Feasibility of Specialized Ability Stealing for Large Language Code Models

Li¹,

Wang²,

Ma³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent progress in large language code models (LLCMs) has led to a dramatic surge in the use of software development. Nevertheless, it is widely known that training a well-performed LLCM requires a plethora of workforce for collecting the data and high quality annotation. Additionally, the training dataset may be proprietary (or partially open source to the public), and the training process is often conducted on a large-scale cluster of GPUs with high costs. Inspired by the recent success of imitation attacks in extracting computer vision and natural language models, this work launches the first imitation attack on LLCMs: by querying a target LLCM with carefully-designed queries and collecting the outputs, the adversary can train an imitation model that manifests close behavior with the target LLCM. We systematically investigate the effectiveness of launching imitation attacks under different query schemes and different LLCM tasks. We also design novel methods to polish the LLCM outputs, resulting in an effective imitation training process. We summarize our findings and provide lessons harvested in this study that can help better depict the attack surface of LLCMs. Our research contributes to the growing body of knowledge on imitation attacks and defenses in deep neural models, particularly in the domain of code related tasks.

show abstract

Section: Evaluation Metricsmentioning

confidence: 99%