Word-level Textual Adversarial Attacking as Combinatorial Optimization

Abstract

Adversarial attacks are carried out to reveal the vulnerability of deep neural networks. Textual adversarial attacking is challenging because text is discrete and a small perturbation can bring significant change to the original input. Word-level attacking, which can be regarded as a combinatorial optimization problem, is a well-studied class of textual attack methods. However, existing word-level attack models are far from perfect, largely because unsuitable search space reduction methods and inefficient optimization algorithms are employed. In this paper, we propose a novel attack model, which incorporates the sememebased word substitution method and particle swarm optimization-based search algorithm to solve the two problems separately. We conduct exhaustive experiments to evaluate our attack model by attacking BiLSTM and BERT on three benchmark datasets. Experimental results demonstrate that our model consistently achieves much higher attack success rates and crafts more high-quality adversarial examples as compared to baseline methods. Also, further experiments show our model has higher transferability and can bring more robustness enhancement to victim models by adversarial training. All the code and data of this paper can be obtained on https://github.com/ thunlp/SememePSO-Attack. * Indicates equal contribution. Yuan developed the method, designed and conducted most experiments; Fanchao formalized the task, designed some experiments and wrote the paper; Chenghao made the original research proposal, performed human evaluation and conducted some experiments.

Explainable AI: A Review of Machine Learning Interpretability Methods

“…Unlike most work on textual adversarial examples, Morpheus produces its adversaries by exploiting the morphology of the text. Zang et al [ 165 ] suggested applying word substitutions using the minimum semantic units, called sememes. The assumption was that the sememes of a word are indicative of the word’s meaning and, therefore, words with the same sememes should be good substitutes for each another.…”

TextAttack: Lessons learned in designing Python frameworks for NLP

“…Queries Cache hits Alzantot et al (2018) 1029 736 Zang et al (2020) 3745 3080 Table 1: "Queries" stands for average number of queries to victim model to attack one sample, while "cache hits" represents the average number of times a query has resulted in a hit to the model output cache. Each cache hit saves a query to the model, so more cache hits indicates a higher performance boost due to caching.…”

“…In some cases, this high-level caching can cause a significant performance increase. We experimented with attacking 100 samples for BERT-base model (Devlin et al, 2018) trained on SST-2 dataset (Socher et al, 2013) using methods proposed by Alzantot et al (2018) and Zang et al (2020). Table 1 shows that in both cases, significant number of queries to the victim model result in hits to the model output cache, helping us save time by avoiding unnecessary computations.…”

“…Deep neural network (DNN) models have seen dominant use in NLP tasks such text classification, natural language inference, machine translation, and question answering. However, despite their state-of-the-art performance, NLP DNNs are still vulnerable to adversarial attacks (Zhang et al, 2020). As a result, there have been growing efforts to develop tools that can help researchers and developers better understand the capability of their NLP models.…”

TextFirewall: Omni-Defending Against Adversarial Texts in Sentiment Classification

“…Researchers have proposed numerous methods to generate adversarial texts, which can be divided into char-level [12], word-level [13], sentence-level [14], and multi-level (i.e., a mixture of the previous three methods) [15,16]. They modify the characters, words, and sentences in the inputs, respectively.…”