Deep Learning Frameworks Applied For Audio-Visual Scene Classification

Pham, Lam; Strobel, Alexander; Schütz, Mina; Lampert, Jasmin; Schlarb, Sven; King, Ross

doi:10.1007/978-3-658-36295-9_6

Cited by 5 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As using ensemble is a rule of thumb to improve the ASC performance and shows effective to deal with the issue of mismatched recording devices [50], [16], [17], [51], [52], [53], [54], we therefore apply an ensemble of multiple spectrogram inputs in this paper. In particular, we use three spectrograms: log-Mel [36], Gammatone (Gam) [55], and Constant Q Transform (CQT) [36].…”

Section: B Further Improve Asc Performance By An Ensemble Of Multiple...mentioning

confidence: 99%

Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context

Pham¹,

Salovic²,

Jalali³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present a comprehensive analysis of Acoustic Scene Classification (ASC), the task of identifying the scene of an audio recording from its acoustic signature.In particular, we firstly propose an inception-based and lowfootprint ASC model, referred to as the ASC baseline. The proposed ASC baseline is then compared with benchmark and high-complexity network architectures of MobileNetV1,

show abstract

Section: B Further Improve Asc Performance By An Ensemble Of Multiple...mentioning

confidence: 99%

Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context

Pham¹,

Salovic²,

Jalali³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Since the predicted probabilities obtained from individual deep neural network architectures can complement each other and a fusion of these predicted probabilities can help to improve the performance, [20][21][22] we propose an ensemble of predicted probabilities in this paper, referred to as PROD late fusion. Let us consider predicted probability of each deep neural network as pn = (p n1 , pn2 , ..., pnC ), where C is the category number and the n th out of N networks evaluated, the predicted probability after PROD fusion p prod = (p 1 , p2 , ..., pC ) is obtained by:…”

Section: Apply An Ensemble To Enhance the Performancementioning

confidence: 99%

Remote sensing image classification using transfer learning- and attention-based deep neural network

Pham

Tran²,

Ngo³

et al. 2022

Image and Signal Processing for Remote Sensing XXVIII

View full text Add to dashboard Cite

The task of remote sensing image scene classification (RSISC), which aims at classifying remote sensing images into groups of semantic categories based on their contents, has assumed an important role in a wide range of applications such as urban planning, natural hazards detection, environmental monitoring, vegetation mapping or geospatial object detection. During the past years, the research community focusing on RSISC tasks has shown significant effort to publish diverse datasets as well as to propose different approaches. Recently, almost all proposed RSISC systems are based on deep learning models, which proves powerful and outperform traditional approaches using image processing and machine learning. In this paper, we also leverage the power of deep learning technologies, evaluate a variety of deep neural network architectures and indicate main factors affecting the performance of a RSISC system. Given the comprehensive analysis, we propose a deep learning based framework for RSISC, which makes use of a transfer learning technique and a multihead attention scheme. The proposed deep learning framework is evaluated on the NWPU-RESISC45 benchmark dataset and achieves a classification accuracy of up to 92.6% and 94.7% with two official data split suggestions (10% and 20% of entire the NWPU-RESISC45 dataset for training). The achieved results are very competitive and show potential for real-life applications.

show abstract

“…Inspired by this simple observation, an increasing number of studies expect to jointly model audio-visual information within scenes. Recent works [13] [14] show that the joint learning of acoustic and visual features can bring additional benefits to AVSC. To exploit the audio-visual information simultaneously, a multi-modal system based on convolutional recurrent neural networks (CRNN) is presented in [15].…”

Section: Introductionmentioning

confidence: 99%

Audio-visual scene classification via contrastive event-object alignment and semantic-based fusion

Hou

Kang

Botteldooren

2022

2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP)

View full text Add to dashboard Cite

Previous works on scene classification are mainly based on audio or visual signals, while humans perceive the environmental scenes through multiple senses. Recent studies on audio-visual scene classification separately fine-tune the largescale audio and image pre-trained models on the target dataset, then either fuse the intermediate representations of the audio model and the visual model, or fuse the coarse-grained decision of both models at the clip level. Such methods ignore the detailed audio events and visual objects in audio-visual scenes (AVS), while humans often identify a scene through both audio events and visual objects within, and the congruence between them. To exploit the fine-grained information of audio events and visual objects in AVS, and coordinate the implicit relationship between audio events and visual objects, this paper proposes a multibranch model equipped with contrastive event-object alignment (CEOA) and semantic-based fusion (SF) for AVSC. CEOA aims to align the learned embeddings of audio events and visual objects by comparing the difference between audio-visual event-object pairs. Then, visual objects associated with certain audio events and vice versa are accentuated by cross-attention and undergo SF for semantic-level fusion. Experiments show that: 1) the proposed AVSC model equipped with CEOA and SF outperforms the results of audio-only and visual-only models, i.e., the audio-visual results are better than the results from a single modality. 2) CEOA aligns the embeddings of audio events and related visual objects on a fine-grained level, and the SF effectively integrates both; 3) Compared with other large-scale integrated systems, the proposed model shows competitive performance, even without using additional datasets and data augmentation tricks.

show abstract

Deep Learning Frameworks Applied For Audio-Visual Scene Classification

Cited by 5 publications

References 21 publications

Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context

Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context

Remote sensing image classification using transfer learning- and attention-based deep neural network

Audio-visual scene classification via contrastive event-object alignment and semantic-based fusion

Contact Info

Product

Resources

About