Spatio-Temporal Attention Pooling for Audio Scene Classification

Phan, Huy; Chén, Oliver Y.; Pham, Lam; Koch, Philipp; Vos, Maarten De; McLoughlin, Ian; Mertins, Alfred

doi:10.21437/interspeech.2019-3040

Cited by 25 publications

(25 citation statements)

References 21 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the frequency domain characteristics of spectrogram features remains static over different time frames, we choose to process temporal and frequency domains separately as proposed in [ 36 ] instead of jointly processing them together as an image. Through this approach, we extract global temporal and frequency attention vectors, e.g.,

and

, and generate the final attention map

, according to

…”

Section: Proposed Temporal-frequency Attention Based Classification Frameworkmentioning

confidence: 99%

High Accurate Environmental Sound Classification: Sub-Spectrogram Segmentation versus Temporal-Frequency Attention Mechanism

Qiao

Zhang

Cao

et al. 2021

Sensors

View full text Add to dashboard Cite

In the important and challenging field of environmental sound classification (ESC), a crucial and even decisive factor is the feature representation ability, which can directly affect the accuracy of classification. Therefore, the classification performance often depends to a large extent on whether the effective representative features can be extracted from the environmental sound. In this paper, we firstly propose a sub-spectrogram segmentation with score level fusion based ESC classification framework, and we adopt the proposed convolutional recurrent neural network (CRNN) for improving the classification accuracy. By evaluating numerous truncation schemes, we numerically figure out the optimal number of sub-spectrograms and the corresponding band ranges, and, on this basis, we propose a joint attention mechanism with temporal and frequency attention mechanisms and use the global attention mechanism when generating the attention map. Finally, the numerical results show that the two frameworks we proposed can achieve 82.1% and 86.4% classification accuracy on the public environmental sound dataset ESC-50, respectively, which is equivalent to more than 13.5% improvement over the traditional baseline scheme.

show abstract

and

, and generate the final attention map

, according to

…”

Section: Proposed Temporal-frequency Attention Based Classification Frameworkmentioning

confidence: 99%

High Accurate Environmental Sound Classification: Sub-Spectrogram Segmentation versus Temporal-Frequency Attention Mechanism

Qiao

Zhang

Cao

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…The sequence of recurrent outputs is then reduced to a feature vector (i.e. view-specific embedding) via spatio-temporal attention pooling suggested in [8]. For classification purpose, the 2D CRNNs make use of two fullyconnected layers with ReLU activation, followed by a final output layer with softmax.…”

Section: Network Architecturementioning

confidence: 99%

“…Good embeddings are crucial for machine learning tasks [1,2,3]. For audio and music classification, in particular, such an embedding can be learned from a variety of low-level features which have been developed alongside the development of the research field, such as Mel-scaled spectrogram [4,5,6,7], Gammatone spectrogram [8,9,2], Constant-Q transform (CQT) spectrogram [10,11,12], and even raw waveform [13,14]. Oftentimes, recognition results obtained from embeddings learned from different low-level inputs vary in the sense that one embedding is good for some target classes while another is good for some other target classes.…”

Section: Introductionmentioning

confidence: 99%

Multi-View Audio And Music Classification

Phan

Nguyêݱn²,

Chén

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

We propose in this work a multi-view learning approach for audio and music classification. Considering four typical low-level representations (i.e. different views) commonly used for audio and music recognition tasks, the proposed multi-view network consists of four subnetworks, each handling one input types. The learned embedding in the subnetworks are then concatenated to form the multi-view embedding for classification similar to a simple concatenation network. However, apart from the joint classification branch, the network also maintains four classification branches on the single-view embedding of the subnetworks. A novel method is then proposed to keep track of the learning behavior on the classification branches and adapt their weights to proportionally blend their gradients for network training. The weights are adapted in such a way that learning on a branch that is generalizing well will be encouraged whereas learning on a branch that is overfitting will be slowed down. Experiments on three different audio and music classification tasks show that the proposed multi-view network not only outperforms the single-view baselines but also is superior to the multi-view baselines based on concatenation and late fusion.

show abstract

“…Although the translation of local patterns in the time domain has little effect on the classification of sound events, the difference across frequency bands has a significant impact on the performance of sound classification [9]. To capture the information about which parts of the features are more relevant to the sound events, attention mechanisms have been proposed [10][11][12][13][14][15][16][17], especially for weakly labelled data where the timing information about the sound events is not available in the training data. In these methods, temporal attention [11,14] is applied to obtain weights for combining feature vectors at different time steps, however, the importance of different frequency bands is not considered.…”

Section: Introductionmentioning

confidence: 99%

Environmental Sound Classification with Parallel Temporal-Spectral Attention

Wang¹,

Zou²,

Chong³

et al. 2020

Interspeech 2020

View full text Add to dashboard Cite

Convolutional neural networks (CNN) are one of the bestperforming neural network architectures for environmental sound classification (ESC). Recently, temporal attention mechanisms have been used in CNN to capture the useful information from the relevant time frames for audio classification, especially for weakly labelled data where the onset and offset times of the sound events are not applied. In these methods, however, the inherent spectral characteristics and variations are not explicitly exploited when obtaining the deep features. In this paper, we propose a novel parallel temporal-spectral attention mechanism for CNN to learn discriminative sound representations, which enhances the temporal and spectral features by capturing the importance of different time frames and frequency bands. Parallel branches are constructed to allow temporal attention and spectral attention to be applied respectively in order to mitigate interference from the segments without the presence of sound events. The experiments on three environmental sound classification (ESC) datasets and two acoustic scene classification (ASC) datasets show that our method improves the classification performance and also exhibits robustness to noise.

show abstract

Spatio-Temporal Attention Pooling for Audio Scene Classification

Cited by 25 publications

References 21 publications

High Accurate Environmental Sound Classification: Sub-Spectrogram Segmentation versus Temporal-Frequency Attention Mechanism

High Accurate Environmental Sound Classification: Sub-Spectrogram Segmentation versus Temporal-Frequency Attention Mechanism

Multi-View Audio And Music Classification

Environmental Sound Classification with Parallel Temporal-Spectral Attention

Contact Info

Product

Resources

About