Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models

Abstract

Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack. Victim models can maintain competitive performance on clean samples while behaving abnormally on samples with a specific trigger word inserted. Previous backdoor attacking methods usually assume that attackers have a certain degree of data knowledge, either the dataset which users would use or proxy datasets for a similar task, for implementing the data poisoning procedure. However, in this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector, with almost no accuracy sacrificed on clean samples. Experimental results on sentiment analysis and sentence-pair classification tasks show that our method is more efficient and stealthier. We hope this work can raise the awareness of such a critical security risk hidden in the embedding layers of NLP models. Our code

RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models

“…Thus, it is a sentence-level attack. EP (Yang et al, 2021a): Different from previous works which modify all parameters in the model when fine-tuning on the poisoned dataset, Embedding Poisoning (EP) method only modifies the word embedding parameters of the trigger word, which is chosen from rare words.…”

“…Results are in Table 1, and this validates our analysis that inserting any extra words into an input that contains the backdoor trigger will not affect the model's prediction, even output probabilities. Therefore, Table 1: The attack success rates (%) of two backdoored models (BadNet (Gu et al, 2017) and EP (Yang et al, 2021a)) trained on Amazon (Blitzer et al, 2007) dataset. Poisoned test samples are constructed by using sentences in the original dataset, sentences from WikiText-103 (Merity et al, 2017) or sentences made up of random words.…”

“…Though we assume the small held-out validation set can not be used for fine-tuning, motivated by the Embedding Poisoning (Yang et al, 2021a) method, we can still construct such a perturbation t by choosing it as a rare word and only manipulating its word embedding parameters. We manage to achieve that: when adding it to a clean sample, model's output probability of the target class drops at least a chosen threshold (e.g., 0.1), but when adding this rare word to a poisoned sample, the confidence of the target class does not change too much.…”

“…Current backdoor attacking researches in natural language process (NLP) (Dai et al, 2019;Garg et al, 2020;Yang et al, 2021a) have shown that the backdoor injected in the model can be triggered by attackers with nearly no failures, and the backdoor effect can be strongly maintained even after the model is further finetuned on a clean dataset (Kurita et al, 2020;Zhang et al, 2021). Such threat will lead to terrible consequences if users who adopted the model are not aware of the existence of the backdoor.…”

“…Following this line, other stealthy and effective attacking methods (Liu et al, 2018b;Nguyen and Tran, 2020;Saha et al, 2020;Zhao et al, 2020) are proposed for hacking image classification models. As for backdoor attacking in NLP, attackers usually use a rare word Garg et al, 2020;Yang et al, 2021a) as the trigger word for data poisoning, or choose the trigger as a long neutral sentence (Dai et al, 2019;Sun, 2020;Yang et al, 2021b). Besides using static and naively chosen triggers, and Chan et al (2020) also make efforts to implement context-aware attacks.…”