All Your Fake Detector are Belong to Us: Evaluating Adversarial Robustness of Fake-News Detectors Under Black-Box Settings

Abstract

With the hyperconnectivity and ubiquity of the Internet, the fake news problem now presents a greater threat than ever before. One promising solution for countering this threat is to leverage deep learning (DL)-based text classification methods for fake-news detection. However, since such methods have been shown to be vulnerable to adversarial attacks, the integrity and security of DL-based fake news classifiers are under question. Although many works study text classification under the adversarial threat, to the best of our knowledge, we do not find any work in literature that specifically analyzes the performance of DL-based fake-news detectors under adversarial settings. We bridge this gap by evaluating the performance of fake-news detectors under various configurations under black-box settings. In particular, we investigate the robustness of four different DL architectural choices-multilayer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN) and a recently proposed Hybrid CNN-RNN trained on three different state-of-the-art datasets-under different adversarial attacks (Text Bugger, Text Fooler, PWWS, and Deep Word Bug) implemented using the state-of-the-art NLP attack library, Text-Attack. Additionally, we explore how changing the detector complexity, the input sequence length, and the training loss affect the robustness of the learned model. Our experiments suggest that RNNs are robust as compared to other architectures. Further, we show that increasing the input sequence length generally increases the detector's robustness. Our evaluations provide key insights to robustify fake-news detectors against adversarial attacks.INDEX TERMS fake news detection, deep neural networks, adversarial attacks, adversarial robustness.

Consistent Valid Physically-Realizable Adversarial Attack against Crowd-flow Prediction Models

“…This suggests that STResnet-2 model is relatively more vulnerable to adversarial perturbations, which appears surprising, as we have observed that STResnet-2 model gives comparatively better performance on the unperturbed dataset, D test , as compared to other architectures. These observations hint at the possibility of an accuracy-robustness tradeoff in the crowdflow prediction models, as has been commonly observed in other DL models [5], [61].…”

“…In their recent work, Jiang et al [3] present TaxiBJ dataset for the year 2021, and use a simple MLP model to benchmark their results. We choose the MLP model motivated by its recency, simplicity, and adversarial transferability-recent works have shown that compared to other architectures, adversarial inputs generated against the MLP models are comparatively general and more effectively transfer to different architectures [5]. STResnet architecture.…”

“…Adversarial attacks are small imperceptible changes to the input to fool DNNs. Since they were discovered by Szegedy et al [44] for image classification, many works have shown that DNNs are generally vulnerable to these attacks in a range of applications including Computer Vision (CV) [7], [8], audio processing [45], networking [46]- [48] and Natual Language Processing(NLP) [5], [49]. Adversarial attacks may assume a black-box [7], [50], [51] or a white-box threat model [52]- [54].…”

A cooperative deep learning model for fake news detection in online social networks

“…Researchers have developed a wide variety of deep learning-based fake news detection systems. For instance, the work by Ni et al ( 2021 ), Ali et al ( 2021 ) and Verma et al ( 2021 ) validate the use of multiple views attention networks (MVNN), adversarial networks, and word embedding feature extraction for fake news identification respectively. These models can achieve an accuracy of over 80% for fake news detection, which makes them deployable for moderately sized systems.…”

Fake news detection: a systematic literature review of machine learning algorithms and datasets

“…In this scenario, Ali et al (2021) express the idea of hyperconnectivity, which consists of the high connection of users and machines, high transmission speed, ease of communication, and access to real-time information worldwide. It is a vibrant path in technological evolution, full of opportunities as well as struggles, such as facilitating fake news.…”