Convolutional Neural Networks Using Log Mel-Spectrogram Separation for Audio Event Classification with Unknown Devices

Abstract

Audio event classification refers to the detection and classification of non-verbal signals, such as dog and horn sounds included in audio data, by a computer. Recently, deep neural network technology has been applied to audio event classification, exhibiting higher performance when compared to existing models. Among them, a convolutional neural network (CNN)-based training method that receives audio in the form of a spectrogram, which is a two-dimensional image, has been widely used. However, audio event classification has poor performance on test data when it is recorded by a device (unknown device) different from that used to record training data (known device). This is because the frequency range emphasized is different for each device used during recording, and the shapes of the resulting spectrograms generated by known devices and those generated by unknown devices differ. In this study, to improve the performance of the event classification system, a CNN based on the log mel-spectrogram separation technique was applied to the event classification system, and the performance of unknown devices was evaluated. The system can classify 16 types of audio signals. It receives audio data at 0.4-s length, and measures the accuracy of test data generated from unknown devices with a model trained via training data generated from known devices. The experiment showed that the performance compared to the baseline exhibited a relative improvement of up to 37.33%, from 63.63% to 73.33% based on Google Pixel, and from 47.42% to 65.12% based on the LG V50.

Deep Neural Networks-based Classification Methodologies of Speech, Audio and Music, and its Integration for Audio Metadata Tagging

“…In this experiment, vanilla MobileNet v2 was less accurate than VGG-Resnet, although the MobileNet v2 model performed better after data augmentation. VGG-Resnet had a comparable performance in the audio event classification domain [32]. MobileNet v2 with time and frequency masking was considerably more accurate than the VGG-Resnet model.…”

“…There were five versions of the AEC model: The results were compared with those for VGG-Resnet, which is an ensemble version of a CNN-based VGG network (VGGnet) [32] and a residual network [33].…”

“…CNNs are widely used to model audio events from audio feature vectors. This type of neural network was mainly used to transform acoustic feature vectors into spectrograms and then to train them [32]. The correlations between local information and feature vectors are learned by the CNN.…”

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

“…The problem caused by differences in frequency‐domain emphasis by various audio input devices was addressed in this study's audio preprocessing step using a log‐Mel spectrogram because it can provide more accurate and detailed characteristics in the high‐ and low‐frequency domains than the Mel‐spectrogram [48]. In addition, log‐Mel spectrograms can improve performance, as demonstrated by the DCASE 2020 challenge for audio scene classification [48]. At a sampling rate of 16000, each 3‐s sample was processed to create a single log‐Mel time‐frequency spectrogram.…”

An Efficient Approach for Securing Audio Data in AI Training with Fully Homomorphic Encryption

“…CNNs excel in classifying audio signals across a spectrum of categories, encompassing speech, music, and environmental sounds. Their proficiency extends to tasks such as speech recognition, speaker identification, and even emotion recognition [18]- [22]. On the other hand, RNNs demonstrate prowess in audio classification and segmentation, effectively disassembling and categorizing audio data with remarkable accuracy [23]- [27].…”