Towards Learning Fine-Grained Disentangled Representations from Speech

Gong, Yuan; Poellabauer, Christian

doi:10.48550/arxiv.1808.02939

Cited by 3 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This has been applied to disentanglement of image features [10], sensor anonymization [11], voice conversion [12,13] and music translation [14]. This idea has also been proposed for disentangling multiple speech attributes by chaining several autoencoders trained in this fashion [15]. However, this adversarial training adds complexity and potential instability, and the necessary labels are difficult to obtain in the audio domain.…”

Section: Related Workmentioning

confidence: 99%

Disentangling speech from surroundings in a neural audio codec

Omran¹,

Zeghidour²,

Borsos³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a method to separate speech signals from noisy environments in the compressed domain of a neural audio codec. We introduce a new training procedure that allows our model to produce structured encodings of audio waveforms given by embedding vectors, where one part of the embedding vector represents the speech signal, and the rest represents the environment. We achieve this by partitioning the embeddings of different input waveforms and training the model to faithfully reconstruct audio from mixed partitions, thereby ensuring each partition encodes a separate audio attribute. As use cases, we demonstrate the separation of speech from background noise or from reverberation characteristics. Our method also allows for targeted adjustments of the audio output characteristics.

show abstract

Section: Related Workmentioning

confidence: 99%

Disentangling speech from surroundings in a neural audio codec

Omran¹,

Zeghidour²,

Borsos³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Natural speech has very complex manifolds [346]) and inherently contains information about the message, gender, age, health status, personality, friendliness, mood, and emotion. All of this information is entangled together [347], and the disentanglement of these attributes in some latent space is a very difficult task that requires extensive training. Most importantly, the training of unsupervised representation learning models is much more difficult in contrast to supervised ones.…”

Section: A Challenge Of Training Deep Architecturesmentioning

confidence: 99%

Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends

Latif,

Rana,

Khalifa

et al. 2020

Preprint

View full text Add to dashboard Cite

Research on speech processing has traditionally considered the task of designing hand-engineered acoustic features (feature engineering) as a separate distinct problem from the task of designing efficient machine learning (ML) models to make prediction and classification decisions. There are two main drawbacks to this approach: firstly, the feature engineering being manual is cumbersome and requires human knowledge; and secondly, the designed features might not be best for the objective at hand. This has motivated the adoption of a recent trend in speech community towards utilisation of representation learning techniques, which can learn an intermediate representation of the input signal automatically that better suits the task at hand and hence lead to improved performance. The significance of representation learning has increased with advances in deep learning (DL), where the representations are more useful and less dependent on human knowledge, making it very conducive for tasks like classification, prediction, etc. The main contribution of this paper is to present an up-to-date and comprehensive survey on different techniques of speech representation learning by bringing together the scattered research across three distinct research areas including Automatic Speech Recognition (ASR), Speaker Recognition (SR), and Speaker Emotion Recognition (SER). Recent reviews in speech have been conducted for ASR, SR, and SER, however, none of these has focused on the representation learning from speech-a gap that our survey aims to bridge. We also highlight different challenges and the key characteristics of representation learning models and discuss important recent advancements and point out future trends. Our review can be used by the speech research community as an essential resource to quickly grasp the current progress in representation learning and can also act as a guide for navigating study in this area of research.

show abstract

“…phoneme and linguistically irrelevant information like speaker characteristics. In the case of speech processing, an ideal disentangled representation would be able to separate fine-grained factors such as speaker identity, noise, recording channels, and prosody [22], as well as the linguistic content. Thus, disentanglement will allow learning of salient and robust representations from the speech that are essential for applications including speech recognition [64], prosody transfer [77,86], speaker verification [66], speech synthesis [31,77], and voice conversion [32], among other applications.…”

Section: Learning Disentangled Representationmentioning

confidence: 99%

Privacy-preserving Voice Analysis via Disentangled Representations

Aloufi,

Haddadi,

Boyle

2020

Preprint

View full text Add to dashboard Cite

Voice User Interfaces (VUIs) are increasingly popular and built into smartphones, home assistants, and Internet of Things (IoT) devices. Despite offering an always-on convenient user experience, VUIs raise new security and privacy concerns for their users. In this paper, we focus on attribute inference attacks in the speech domain, demonstrating the potential for an attacker to accurately infer a target user's sensitive and private attributes (e.g. their emotion, sex, or health status) from deep acoustic models. To defend against this class of attacks, we design, implement, and evaluate a user-configurable, privacy-aware framework for optimizing speechrelated data sharing mechanisms. Our objective is to enable primary tasks such as speech recognition and user identification, while removing sensitive attributes in the raw speech data before sharing it with a cloud service provider. We leverage disentangled representation learning to explicitly learn independent factors in the raw data. Based on a user's preferences, a supervision signal informs the filtering out of invariant factors while retaining the factors reflected in the selected preference. Our experimental evaluation over five datasets shows that the proposed framework can effectively defend against attribute inference attacks by reducing their success rates to approximately that of guessing at random, while maintaining accuracy in excess of 99% for the tasks of interest. We conclude that negotiable privacy settings enabled by disentangled representations can bring new opportunities for privacy-preserving applications.

show abstract

Towards Learning Fine-Grained Disentangled Representations from Speech

Cited by 3 publications

References 16 publications

Disentangling speech from surroundings in a neural audio codec

Disentangling speech from surroundings in a neural audio codec

Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends

Privacy-preserving Voice Analysis via Disentangled Representations

Contact Info

Product

Resources

About