Audio-Visual Model Distillation Using Acoustic Images

Abstract

In this paper, we investigate how to learn rich and robust feature representations for audio classification from visual data and acoustic images, a novel audio data modality. Former models learn audio representations from raw signals or spectral data acquired by a single microphone, with remarkable results in classification and retrieval. However, such representations are not so robust towards variable environmental sound conditions. We tackle this drawback by exploiting a new multimodal labeled action recognition dataset acquired by a hybrid audio-visual sensor that provides RGB video, raw audio signals, and spatialized acoustic data, also known as acoustic images, where the visual and acoustic images are aligned in space and synchronized in time. Using this richer information, we train audio deep learning models in a teacher-student fashion. In particular, we distill knowledge into audio networks from both visual and acoustic image teachers. Our experiments suggest that the learned representations are more powerful and have better generalization capabilities than the features learned from models trained using just single-microphone audio data.

Knowledge Distillation: A Survey

“…To satisfy these requirements, knowledge distillation is widely studied and applied in many speech recognition tasks. There are many knowledge distillation systems for designing lightweight deep acoustic models for speech recognition (Chebotar and Waters, 2016;Wong and Gales, 2016;Chan et al, 2015;Price et al, 2016;Fukuda et al, 2017;Bai et al, 2019b;Ng et al, 2018;Albanie et al, 2018;Lu et al, 2017;Shi et al, 2019a;Roheda et al, 2018;Shi et al, 2019b;Gao et al, 2019;Ghorbani et al, 2018;Takashima et al, 2018;Watanabe et al, 2017;Shi et al, 2019c;Asami et al, 2017;Huang et al, 2018;Shen et al, 2018;Perez et al, 2020;Shen et al, 2019a;Oord et al, 2018). In particular, these KD-based speech recognition applications have spoken language identification (Shen et al, 2018(Shen et al, , 2019a), text-independent speaker recognition (Ng et al, 2018), audio classification (Gao et al, 2019;Perez et al, 2020), speech enhancement (Watanabe et al, 2017), acoustic event detection (Price et al, 2016;Shi et al, 2019a,b), speech synthesis (Oord et al, 2018) and so on.…”

“…Most existing knowledge distillation methods for speech recognition, use teacher-student architectures to improve the efficiency and recognition accuracy of acoustic models (Chan et al, 2015;Chebotar and Waters, 2016;Lu et al, 2017;Price et al, 2016;Shen et al, 2018;Gao et al, 2019;Shen et al, 2019a;Shi et al, 2019c,a;Watanabe et al, 2017;Perez et al, 2020). Using a recurrent neural network (RNN) for holding the temporal information from speech sequences, the knowledge from the teacher RNN acoustic model was transferred into a small student DNN model (Chan et al, 2015).…”

Uncertainty Aware Review Hallucination for Science Article Classification

“…c) A model that aims to support the editorial decision process should only assume the availability of human review text during training, and be able to make recommendations in their absence. Inspired by missing modality hallucination methods (Hoffman et al, 2016;Tang et al;Pérez et al, 2020)), we propose a realistic system that uses all available data for training, but imputes review representations at test time based on the abstract text.…”

“…Figure1depicts an overview of our architecture. Inspired by modality hallucination studies(Hoffman et al, 2016;Pérez et al, 2020), we use the abstract module to predict both the abstract h abs i…”

Human Action Recognition from Various Data Modalities: A Review

“…Alwassel et al [399] proposed a self-supervised method, called Cross-Modal Deep Clustering (XDC), to utilize the semantic correlation and the differences between RGB and audio modalities. In the work of [400], audio deep learning models were trained, and the visual and acoustic images were exploited in a teacher-student fashion.…”