PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

Abstract

Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification and sound event detection. Recently neural networks have been applied to solve audio pattern recognition problems. However, previous systems focus on small datasets, which limits the performance of audio pattern recognition systems. Recently in computer vision and natural language processing, systems pretrained on large datasets have generalized well to several tasks. However, there is limited research on pretraining neural networks on large datasets for audio pattern recognition. In this paper, we propose large-scale pretrained audio neural networks (PANNs) trained on AudioSet. We propose to use Wavegram, a feature learned from waveform, and the mel spectrogram as input. We investigate the performance and complexity of a variety of convolutional neural networks. Our proposed AudioSet tagging system achieves a state-of-the-art mean average precision (mAP) of 0.439, outperforming the best previous system of 0.392. We transferred a PANN to six audio pattern recognition tasks and achieve state-of-the-art performance in many tasks. Source code and pretrained models have been released.

Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

“…b. Audio Modality: Following the aforementioned procedure of our visual modality comparison, Table 4 presents the results of using pre-trained audio features [25] compared with audio features learned using ShotCoL on Ad-Cuepoints dataset. Results using different number of shots are presented showing that ShotCoL is able to outperform existing approach by a sizable margin, demonstrating its effectiveness on audio modality.…”

“…To extract the audio embedding from each shot, we use a Wavegram-Logmel CNN [25] which incorporates a 14-layer CNN similar in architecture to the VGG [17] network. We sample 10-second mono audio samples at a rate of 32 kHz from each shot.…”

“…For shots longer than 10 seconds, we extract a 10-second window from the center. These inputs are provided to the Wavegram-Logmel network [25] to extract a 2048-dimensional feature vector for each shot.…”

Distilling Audio-Visual Knowledge by Compositional Contrastive Learning

“…We use R(2+1)D-18 [58] as the video student network. The audio and image teacher networks are the 1D-CNN14 [35] and 2D-ResNet34 [28], pretrained on the AudioSet [22] and ImageNet [15]. The model weights of the teacher networks are kept frozen during training.…”

Weakly-Supervised Classification and Detection of Bird Sounds in the Wild. A BirdCLEF 2021 Solution

“…[5] and Lasseck [6,7] introduced deep learning techniques for the "Bird species identification in soundscapes" problem. State-of-the-art solutions are based on Deep Convolutional Neural Networks (CNNs) [8,9,10], usually, deep CNNs with attention mechanisms are selected as backbone in these experiments [11,12,13,14,15]. Pretrained audio neural networks (PANNs) [14] provide a multi-task state-of-the-art baseline for audio related tasks, in previous competitions these networks proved their generalization capability.…”