Deep Learning for Black-Box Modeling of Audio Effects

Abstract

Virtual analog modeling of audio effects consists of emulating the sound of an audio processor reference device. This digital simulation is normally done by designing mathematical models of these systems. It is often difficult because it seeks to accurately model all components within the effect unit, which usually contains various nonlinearities and time-varying components. Most existing methods for audio effects modeling are either simplified or optimized to a very specific circuit or type of audio effect and cannot be efficiently translated to other types of audio effects. Recently, deep neural networks have been explored as black-box modeling strategies to solve this task, i.e., by using only input–output measurements. We analyse different state-of-the-art deep learning models based on convolutional and recurrent neural networks, feedforward WaveNet architectures and we also introduce a new model based on the combination of the aforementioned models. Through objective perceptual-based metrics and subjective listening tests we explore the performance of these models when modeling various analog audio effects. Thus, we show virtual analog models of nonlinear effects, such as a tube preamplifier; nonlinear effects with memory, such as a transistor-based limiter and nonlinear time-varying effects, such as the rotating horn and rotating woofer of a Leslie speaker cabinet.

Special Issue on Digital Audio Effects

“…Deep Learning for Black-Box Modeling of Audio Effects by Marco A. Martínez Ramírez, Emmanouil Benetos and Joshua D. Reiss [6] testifies the disruption of the traditional analog model-based approach to DAFx design, in favor of black-box strategies based on neural networks. The authors analyse different state-of-the-art deep learning models and explore their objective and subjective performance in various analog audio effect case studies.…”

A Deep Learning Approach to Intelligent Drum Mixing With the Wave-U-Net

“…To avoid the above issues, novel time-domain approaches have been proposed in recent literature. Martínez Ramírez et al [24,25,26,27] investigated DNNs for audio processing tasks, such as modeling of various types of audio effects. Similarly, Wright et al [28] explored variants of the WaveNet architecture [29] and recurrent neural networks to model distortion audio effects.…”

“…Training is performed as in the original paper [7], but using the L1 instead of the L2 distance as training loss, based on previous observations with neural models that output raw audio obtaining perceptually more convincing results [24,26]. Adam is used as optimizer and we use an early stopping patience of 20 epochs followed by a finetuning step.…”

Differentiable Signal Processing With Black-Box Audio Effects

“…The most common approach for the latter is adaptive audio effects or signal processing systems based on the modeling and automation of traditional processors [4]. More recent deep learning methods for audio effects modeling and intelligent audio effects include 1) end-to-end direct transformation methods [5,6,7], where a neural proxy learns and applies the transformation of an audio effect target 2) parameter estimators, where a deep neural network (DNN) predicts the parameter settings of an audio effect [8,9,10] and 3) differentiable digital signal processing (DDSP) [11,12], where signal processing structures are implemented within a deep learning auto-differentiation framework and trained via backpropagation.…”

“…First, direct transform approaches can require special, custom modeling strategies per effect (e.g. distortion), are often based on large and expensive networks, and/or use models with limited or no editable parameter control [5,6]. Second, methods with Fig.…”