Deep Learning Frameworks Applied For Audio-Visual Scene Classification

Abstract

In this paper, we present deep learning frameworks for audio-visual scene classification (SC) and indicate how individual visual and audio features as well as their combination affect SC performance. Our extensive experiments, which are conducted on DCASE (IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events) Task 1B development dataset, achieve the best classification accuracy of 82.2%, 91.1%, and 93.9% with audio input only, visual input only, and both audio-visual input, respectively. The highest classification accuracy of 93.9%, obtained from an ensemble of audio-based and visual-based frameworks, shows an improvement of 16.5% compared with DCASE baseline.

A Low-Complexity Deep Learning Framework For Acoustic Scene Classification

“…To deal with the ASC challenge of mismatched recording devices, the state-of-the-art systems mainly leverage ensemble techniques: Ensemble of spectrogram inputs [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12] or ensemble of different classification models [13], [14], [15]. Although these approaches prove effective to deal with the issue of mismatched recording devices and achieve potential results, they present large model complexity.…”

Low-complexity deep learning frameworks for acoustic scene classification

“…To deal with one of the main ASC challenges, mismatched recording devices, a variety of methods have been proposed, which mainly make use of ensemble techniques: Ensemble of spectrogram inputs [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11] (i.e., This approach uses multiple spectrogram inputs but only one model architecture) or ensemble of different classification models [12], [13], [14] (i.e., This approach uses only one spectrogram input, but explores the spectrogram by different model architectures). Although these approaches help to achieve good results, they show very large footprint models, which causes challenging to apply on edge-devices.…”