Target Language Extraction at Multilingual Cocktail Parties

Borsdorf, Marvin; Li, Haizhou; Schultz, Tanja

doi:10.1109/asru51503.2021.9688052

Cited by 6 publications

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reproducing a similar mechanism would require TSE systems that operate with semantic clues, which introduces novel challenges concerning how to represent semantic information and exploit it within a TSE system. Some works have started to explore this direction, such as conditioning on languages [59] or more abstract concepts [60].…”

Section: Exploring Other Cluesmentioning

confidence: 99%

Neural Target Speech Extraction: An Overview

Žmolíková¹,

Delcroix²,

Ochiai³

et al. 2023

Preprint

View full text Add to dashboard Cite

Humans can listen to a target speaker even in challenging acoustic conditions that have noise, reverberation, and interfering speakers. This phenomenon is known as the cocktail-party effect. For decades, researchers have focused on approaching the listening ability of humans. One critical issue is handling interfering speakers because the target and non-target speech signals share similar characteristics, complicating their discrimination. Target speech/speaker extraction (TSE) isolates the speech signal of a target speaker from a mixture of several speakers with or without noises and reverberations using clues that identify the speaker in the mixture. Such clues might be a spatial clue indicating the direction of the target speaker, a video of the speaker's lips, or a pre-recorded enrollment utterance from which their voice characteristics can be derived. TSE is an emerging field of research that has received increased attention in recent years because it offers a practical approach to the cocktailparty problem and involves such aspects of signal processing as audio, visual, array processing, and deep learning. This paper focuses on recent neural-based approaches and presents an in-depth overview of TSE. We guide readers through the different major approaches, emphasizing the similarities among frameworks and discussing potential future directions.

show abstract

Section: Exploring Other Cluesmentioning

confidence: 99%

Neural Target Speech Extraction: An Overview

Žmolíková¹,

Delcroix²,

Ochiai³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Research on target speech extraction has conventionally focused on extracting the speech of a target speaker in a mixture of overlapping speakers by exploiting physical clues such as pre-recorded enrollment utterances [46], direction information [12], or video [1,9,28,40] to identify the target speaker. Meanwhile, we can use semantic clues, such as language [3,42] or content of speech, to focus our attention on the conversation we want to hear. For example, if our name is mentioned or the topic of a conversation nearby interests us, we turn our attention to that speaker.…”

Section: Introductionmentioning

confidence: 99%

ConceptBeam: Concept Driven Target Speech Extraction

Ohishi,

Delcroix,

Ochiai

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a novel framework for target speech extraction based on semantic information, called ConceptBeam. Target speech extraction means extracting the speech of a target speaker in a mixture. Typical approaches have been exploiting properties of audio signals, such as harmonic structure and direction of arrival. In contrast, Con-ceptBeam tackles the problem with semantic clues. Specifically, we extract the speech of speakers speaking about a concept, i.e., a topic of interest, using a concept specifier such as an image or speech. Solving this novel problem would open the door to innovative applications such as listening systems that focus on a particular topic discussed in a conversation. Unlike keywords, concepts are abstract notions, making it challenging to directly represent a target concept. In our scheme, a concept is encoded as a semantic embedding by mapping the concept specifier to a shared embedding space. This modality-independent space can be built by means of deep metric learning using paired data consisting of images and their spoken captions. We use it to bridge modality-dependent information, i.e., the speech segments in the mixture, and the specified, modalityindependent concept. As a proof of our scheme, we performed experiments using a set of images associated with spoken captions. That is, we generated speech mixtures from these spoken captions and used the images or speech signals as the concept specifiers. We then extracted the target speech using the acoustic characteristics of the identified segments. We compare ConceptBeam with two methods: one based on keywords obtained from recognition systems and another based on sound source separation. We show that ConceptBeam clearly outperforms the baseline methods and effectively extracts speech based on the semantic representation.

show abstract

Experts Versus All-Rounders: Target Language Extraction for Multiple Target Languages

Borsdorf

Scheck

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Target Language Extraction at Multilingual Cocktail Parties

Cited by 6 publications

References 29 publications

Neural Target Speech Extraction: An Overview

Neural Target Speech Extraction: An Overview

ConceptBeam: Concept Driven Target Speech Extraction

Experts Versus All-Rounders: Target Language Extraction for Multiple Target Languages

Contact Info

Product

Resources

About