WaveGlow: A Flow-based Generative Network for Speech Synthesis

Prenger, Ryan; Valle, Rafael F.; Catanzaro, Bryan

doi:10.48550/arxiv.1811.00002

Cited by 17 publications

(36 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from the original spectrogram, deep learning architectures have also experimented with non linear spectrograms such as mel-spectrograms [21] [22] [23] [24] [25] [26] [27] or Constant-Q Transformations (CQT) [28]. The mel-spectrogram is generated by the application of perceptual filters on the DFT called mel-filter bands.…”

Section: B Spectrogramsmentioning

confidence: 99%

“…WaveGlow [21], a flow-based generative network, can synthesise sound from its mel-spectrogram. By applying an Affine Coupling Layer and a 1x1 Invertible Convolution, the model aims to maximise the likelihood of the training data.…”

Section: B Normalizing Flowmentioning

confidence: 99%

“…Furthermore, in order to attract an accountable number of subjects with specific characteristics, Amazon Mechanical Turk has been widely used. In many cases, raters have been asked to pass a hearing test [21], keep headphones on [61] [11], or only native speakers for evaluating speech have been asked [60] [61] [54].…”

Section: A Perceptual Evaluationmentioning

confidence: 99%

“…In these mean opinion score tests, subjects have been asked to rate a sound in a five-point Likert scale in terms of pleasantness [21], naturalness [11] [13] [63], sound quality [17] or speaker diversity [32]. In addition, subjects have been requested to express a preference between sounds of two generative models hearing the same pitch [30] or speech [17] [11].…”

Section: A Perceptual Evaluationmentioning

confidence: 99%

See 3 more Smart Citations

Audio representations for deep learning in sound synthesis: A review

Anastasia¹,

O’Leary²

2022

Preprint

View full text Add to dashboard Cite

The rise of deep learning algorithms has led many researchers to withdraw from using classic signal processing methods for sound generation. Deep learning models have achieved expressive voice synthesis, realistic sound textures, and musical notes from virtual instruments. However, the most suitable deep learning architecture is still under investigation. The choice of architecture is tightly coupled to the audio representations. A sound's original waveform can be too dense and rich for deep learning models to deal with efficientlyand complexity increases training time and computational cost. Also, it does not represent sound in the manner in which it is perceived. Therefore, in many cases, the raw audio has been transformed into a compressed and more meaningful form using upsampling, feature-extraction, or even by adopting a higher level illustration of the waveform. Furthermore, conditional on the form chosen, additional conditioning representations, different model architectures, and numerous metrics for evaluating the reconstructed sound have been investigated. This paper provides an overview of audio representations applied to sound synthesis using deep learning. Additionally, it presents the most significant methods for developing and evaluating a sound synthesis architecture using deep learning models, always depending on the audio representation.

show abstract

Section: B Spectrogramsmentioning

confidence: 99%

Section: B Normalizing Flowmentioning

confidence: 99%

Section: A Perceptual Evaluationmentioning

confidence: 99%

Section: A Perceptual Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Audio representations for deep learning in sound synthesis: A review

Anastasia¹,

O’Leary²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With the recent developments in the NFs architectures [8][9][10], the qualitative and quantitative results as well as computational requirements are extremely competitive. The applications include but are not limited to image generation [8,11], noise modelling [12], video generation [13], audio generation [14][15][16], graph generation [17], reinforcement learning [18][19][20], computer graphics [21] and physics [22][23][24][25][26].…”

Section: Introductionmentioning

confidence: 99%

Out-of-Distribution Detection of Melanoma using Normalizing Flows

Valiuddin¹,

Viviers²

2021

Preprint

View full text Add to dashboard Cite

Generative modelling has been a topic at the forefront of machine learning research for a substantial amount of time. With the recent success in the field of machine learning, especially in deep learning, there has been an increased interest in explainable and interpretable machine learning. The ability to model distributions and provide insight in the density estimation and exact data likelihood is an example of such a feature. Normalizing Flows (NFs), a relatively new research field of generative modelling, has received substantial attention since it is able to do exactly this at a relatively low cost whilst enabling competitive generative results. While the generative abilities of NFs are typically explored, we focus on exploring the data distribution modelling for Out-of-Distribution (OOD) detection. Using one of the state-of-the-art NF models, GLOW, we attempt to detect OOD examples in the ISIC dataset. We notice that this model under performs in conform related research. To improve the OOD detection, we explore the masking methods to inhibit co-adaptation of the coupling layers however find no substantial improvement. Furthermore, we utilize Wavelet Flow which uses wavelets that can filter particular frequency components, thus simplifying the modeling process to data-driven conditional wavelet coefficients instead of complete images. This enables us to efficiently model larger resolution images in the hopes that it would capture more relevant features for OOD. The paper that introduced Wavelet Flow mainly focuses on its ability of sampling high resolution images and did not treat OOD detection. We present the results and propose several ideas for improvement such as controlling frequency components, using different wavelets and using other state-of-the-art NF architectures.

show abstract

Integrated visual transformer and flash attention for lip-to-speech generation GAN

Yang,

Bai,

Liu

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Lip-to-Speech (LTS) generation is an emerging technology that is highly visible, widely supported, and rapidly evolving. LTS has a wide range of promising applications, including assisting speech impairment and improving speech interaction in virtual assistants and robots. However, the technique faces the following challenges: (1) Chinese lip-to-speech generation is poorly recognized. (2) The wide range of variation in lip-speaking is poorly aligned with lip movements. Addressing these challenges will contribute to advancing Lip-to-Speech (LTS) technology, enhancing the communication abilities, and improving the quality of life for individuals with disabilities. Currently, lip-to-speech generation techniques usually employ the GAN architecture but suffer from the following problems: The primary issue lies in the insufficient joint modeling of local and global lip movements, resulting in visual ambiguities and inadequate image representations. To solve these problems, we design Flash Attention GAN (FA-GAN) with the following features: (1) Vision and audio are separately coded, and lip motion is jointly modelled to improve speech recognition accuracy. (2) A multilevel Swin-transformer is introduced to improve image representation. (3) A hierarchical iterative generator is introduced to improve speech generation. (4) A flash attention mechanism is introduced to improve computational efficiency. Many experiments have indicated that FA-GAN can recognize Chinese and English datasets better than existing architectures, especially the recognition error rate of Chinese, which is only 43.19%, the lowest among the same type.

show abstract

WaveGlow: A Flow-based Generative Network for Speech Synthesis

Cited by 17 publications

References 0 publications

Audio representations for deep learning in sound synthesis: A review

Audio representations for deep learning in sound synthesis: A review

Out-of-Distribution Detection of Melanoma using Normalizing Flows

Integrated visual transformer and flash attention for lip-to-speech generation GAN

Contact Info

Product

Resources

About