Learned Token Pruning for Transformers

Anchor pruning for object detection

“…Pruning methods can also differ in the way that token reduction is applied. In fixed rate pruning (Goyal et al, 2020;Rao et al, 2021;Bolya et al, 2023;Liang et al, 2022;Xu et al, 2022) a predefined number of tokens is removed per layer, while in adaptive approaches (Kim et al, 2022;Yin et al, 2021; the tokens are pruned dynamically based on the input.…”

“…Our learned thresholds approach is conceptually similar to learned token pruning as introduced in Kim et al (2022). In each transformer block an importance score is calculated for every token x i , i ∈ {1, ..., n}, where n = hw is the number of tokens 1 .…”

“…The learned token pruning implementation in Kim et al (2022) multiplies its mask with the tokens in order to create zero valued tokens. However, these tokens do not remain zero due to bias terms in MLP layers; furthermore adding zero valued tokens changes attention calculations compared to removing those tokens.…”

“…which can be interpreted as the normalized amount that all tokens x k attend to token x i (Kim et al, 2022).…”

Multilingual Speech Translation with Unified Transformer: Huawei Noah’s Ark Lab at IWSLT 2021

“…Tang et al [28] presents a top-down layer by layer patch slimming algorithm to reduce the computational cost in pre-trained Vision Transformers. The core strategy of these algorithms and other similar works [11,13,19] is to abandon redundant tokens to reduce the computational complexity of the model.…”