nnAudio: An on-the-Fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolutional Neural Networks

Abstract

In this paper, we present nnAudio, a new neural network-based audio processing framework with graphics processing unit (GPU) support that leverages 1D convolutional neural networks to perform time domain to frequency domain conversion. It allows on-the-fly spectrogram extraction due to its fast speed, without the need to store any spectrograms on the disk. Moreover, this approach also allows backpropagation on the waveforms-to-spectrograms transformation layer, and hence, the transformation process can be made trainable, further optimizing the waveform-to-spectrogram transformation for the specific task that the neural network is trained on. All spectrogram implementations scale as Big-O of linear time with respect to the input length. nnAudio, however, leverages the compute unified device architecture (CUDA) of 1D convolutional neural network from PyTorch, its short-time Fourier transform (STFT), Mel spectrogram, and constant-Q transform (CQT) implementations are an order of magnitude faster than other implementations using only the central processing unit (CPU). We tested our framework on three different machines with NVIDIA GPUs, and our framework significantly reduces the spectrogram extraction time from the order of seconds (using a popular python library librosa) to the order of milliseconds, given that the audio recordings are of the same length. When applying nnAudio to variable input audio lengths, an average of 11.5 hours are required to extract 34 spectrogram types with different parameters from the MusicNet dataset using librosa. An average of 2.8 hours is required for nnAudio, which is still four times faster than librosa. Our proposed framework also outperforms existing GPU processing libraries such as Kapre and torchaudio in terms of processing speed.

Audio-to-score singing transcription based on a CRNN-HSMM hybrid model

“…For each signal sampled at kHz, we used a STFT with a Hann window of points and a shifting interval of points ( ms) for calculating the amplitude spectrogram on the logarithmic frequency axis having bins per semitone (i.e. bin per cents) between Hz (C1) and Hz (C7) [30]. We then computed the HCQT-like spectrogram by stacking the -harmonic-shifted versions of the original spectrogram, where (i.e.…”

An End-To-End Emotion Recognition Framework Based on Temporal Aggregation of Multimodal Information

“…where f m−1 , f m , f m+1 are evenly-spaced discrete frequencies in the Mel-scaled frequency domain, whereas the index m ∈ {1, 2, ..., n mel } indicates the number of the filter in the filterbank. The log-Mel spectrogram is obtained by taking the logarithm of the STFT result, multiplied by the filterbanks' coefficients, at each timestep [35]:…”

Web-Based Music Genre Classification for Timeline Song Visualization and Analysis

“…Even though CNNs were initially conceived for artificial vision, they have yielded good results in problems in which visual representations of input data are feasible. When working with audio, CNNs are frequently used in combination with mid-level time-frequency visual representations, known as spectrograms [32]. The Audioset VGGish model is an example of this.…”

“…CNNs normally use audio visual representations (spectrograms) as inputs [32], [62]. After the CNN is trained, it is not uncommon to see how researchers apply a transfer learning approach and extract the embeddings of the final layers from the network to generate pre-trained models [57], [63].…”