Efficient Training of Audio Transformers with Patchout

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

“…Audio Spectrogram Transformers: Inspired by the Vision Transformer (ViT) [12], transformers capable of processing images have been adapted to the audio domain. Vision and Audio Spectrogram transformers [16,17,18,19] extract overlapping patches with a certain stride and size of the input image, add a positional encoding, and apply transformer layers to the flattened sequence of patches. Transformer layers use a global attention mechanism that leads to computation and memory complexity scaling quadratically with with respect to the input sequence.…”

“…We apply the same preprocessing as in [17]. We use mono audio sampled at 32 kHz and compute Mel features from 25 ms windows with a hop size of 10 ms.…”

“…For KD, we ensemble PaSST [17] models with different patch sizes and strides as a teacher. We pre-compute the predictions of 9 different PaSST models 2 on AudioSet to speed up training and form an ensemble by averaging the logits.…”

“…1. Crosses denote models based on Transformer architecture (Audio-MAE [19], HTS-AT [18], PaSST-S [17], PaSST-S-L [17], AST [16], KD-AST [10]) and circles denote models based on CNNs (PSLA [2], ERANN-1-6 [3], Wavegramlogmel-CNN [1], CNN14 [1], KD-CNN [10], MobileNets [7] -ours).…”

“…the art in AT [16,17,18,19]. However, Transformers are complex in terms of parameters compared to CNNs, and the global self-attention mechanism scales quadratically with respect to the sequence length, making training and inference slow, and the deployment on edge devices infeasible.…”