DDSP: Differentiable Digital Signal Processing

Abstract

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available 1 and we welcome further contributions from the community and domain experts.

Audio representations for deep learning in sound synthesis: A review

“…This approach assisted the process of text in deep learning applications by inserting the property of closer embeddings in vector space to encode words with similar meaning. The same approach has been adopted by sound processing to reduce the dimensionality of the signal [40] [22], enhance the timbre synthesis [3] or even generate a more interpretable representation [41] [42] to effectively extract parameters for a synthesizer. In [7] an autoencoder generates a latent representation to condition a WaveNet model while Dhariwal et al [43] implemented three separate encoders to generate vectors with different temporal resolutions.…”

“…However, in cases where the amount of training data is not sufficient, additional data with similar properties can be included by applying conditioning methods. Following these techniques, the generated sound can be conditioned on specific traits such as speaker's voice [47] [27], independent pitch [3] [48] [36], linguistic features [49] [17] or latent representations [4] [45]. Instead of one-hot-embeddings, some implementations have also used a confusion matrix to capture a variation of emotions [39], while others provided supplementary positional information of each segment conditioning music to the artist or genre [43].…”

“…Many variations of VAE have been applied for sound generation topics. In [3] they used VAE with feedforward networks and an additive synthesiser to reproduce monophonic musical notes. In [69] and [40] they applied convolutional layers while in [36] a Variational Parametric Synthesiser was proposed using a conditional VAE.…”

“…Distances-based measurements have also been investigated individually by separate parameter estimations. In [3] distances between the generated loudness and fundamental frequency of synthesised and training data are used.…”

“…These models discover latent representations based on the distribution of the initial data and then sample from this distribution to generate new acoustic signals with the same properties as the original ones. In many cases, the deep learning models can operate along with signal processing algorithms and enhance their expression capabilities [3] [4].…”