Accumulative Poisoning Attacks on Real-time Data

Abstract

Collecting training data from untrusted sources exposes machine learning services to poisoning adversaries, who maliciously manipulate training data to degrade the model accuracy. When trained on offline datasets, poisoning adversaries have to inject the poisoned data in advance before training, and the order of feeding these poisoned batches into the model is stochastic. In contrast, practical systems are more usually trained/fine-tuned on sequentially captured real-time data, in which case poisoning adversaries could dynamically poison each data batch according to the current model state. In this paper, we focus on the real-time settings and propose a new attacking strategy, which affiliates an accumulative phase with poisoning attacks to secretly (i.e., without affecting accuracy) magnify the destructive effect of a (poisoned) trigger batch. By mimicking online learning and federated learning on CIFAR-10, we show that the model accuracy will significantly drop by a single update step on the trigger batch after the accumulative phase. Our work validates that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects, with no need to explore complex techniques. * Equal contribution Preprint. Under review.

A Roadmap for Big Model

“…This type of data poisoning attacks only manipulate training data and labels without the need to modify testing data after the victim model is deployed. Training-only poisoning attacks include both untargeted attacks where the adversary aims to degrade model performance on normal testing data [995,997,998,999], and targeted attacks in which the adversary aims to change the behavior of the model on particular testing inputs [1000,1001,1002]. Below we introduce some typical approaches.…”

“…However, it applies a greedy strategy to lower down model accuracy at each update step, which limits the step-wise destructive effect. Recent work [999] proposes accumulative poisoning attacks, where the model states are secretly (i.e., keeping accuracy in a reasonable range) activated towards a trigger batch by the accumulative phase, and the model is suddenly broken down by feeding in the trigger batch, before the monitor gets conscious of the attacks.…”

Thinking Two Moves Ahead: Anticipating Other Users Improves Backdoor Attacks in Federated Learning

“…Wang et al (2020) note that backdoor success is high in edge cases not seen in training and that backdoors that attack "rare" samples (such as only airplanes in a specific color in images, or a specific sentence in text) can be much more successful, as other users do not influence these predictions significantly. A number of variants of this attack exist (Costa et al, 2021;Pang et al, 2021;Fang et al, 2020;Baruch et al, 2019;Xie et al, 2019;Datta et al, 2021;Yoo & Kwak, 2022;Zhang et al, 2019;Sun et al, 2022), for example allowing for collusion between multiple users or generating additional data for the attacker. In this work we will focus broadly on the threat model of Bagdasaryan et al (2019); Wang et al (2020).…”