Using Complexity-Identical Human- and Machine-Directed Utterances to Investigate Addressee Detection for Spoken Dialogue Systems

Abstract

Human-machine addressee detection (H-M AD) is a modern paralinguistics and dialogue challenge that arises in multiparty conversations between several people and a spoken dialogue system (SDS) since the users may also talk to each other and even to themselves while interacting with the system. The SDS is supposed to determine whether it is being addressed or not. All existing studies on acoustic H-M AD were conducted on corpora designed in such a way that a human addressee and a machine played different dialogue roles. This peculiarity influences speakers’ behaviour and increases vocal differences between human- and machine-directed utterances. In the present study, we consider the Restaurant Booking Corpus (RBC) that consists of complexity-identical human- and machine-directed phone calls and allows us to eliminate most of the factors influencing speakers’ behaviour implicitly. The only remaining factor is the speakers’ explicit awareness of their interlocutor (technical system or human being). Although complexity-identical H-M AD is essentially more challenging than the classical one, we managed to achieve significant improvements using data augmentation (unweighted average recall (UAR) = 0.628) over native listeners (UAR = 0.596) and a baseline classifier presented by the RBC developers (UAR = 0.539).

Acoustic-Based Automatic Addressee Detection for Technical Systems: A Review

“…First, with greater importance on true wake-word independence. In Akhtiamov et al (2020) the classification process was improved by employing an ensemble classifier, consisting of several classification tasks that are combined in a late fusion approach, which allows combining the strength of the different methods into one singular system. Second, with a heightened sense of privacy by changing a system to ignore information that is not directed to the device either by using features with a limited use to detect what has been said (Baumann and Siegert, 2020) or by extending a wakeword detection system by an acoustic feature classification to improve the security of such a system from false activations (Wang et al, 2020).…”

“…For using RBC as a test set, one of the best performances of 60.90% Unweighted Average Recall (UAR) was achieved, when using VACC data together with RBC data with an end-to-end (e2e) speech processing model (Akhtiamov et al, 2019). But using a more complex meta-model, that makes use of different models to combine different layers of information, gives a slightly better performance of 62.80 UAR (Akhtiamov et al, 2019(Akhtiamov et al, , 2020.…”

“…Additionally, some studies also investigated cross-corpus experiments using SVC and VACC (Akhtiamov et al, 2019) and in combination with RBC (Akhtiamov et al, 2020). The authors used all of these three datasets for testing and training.…”

“…When only comparing VACC and SVC, which are designed with the same target classes, this rises to 172 similar features. In Akhtiamov et al (2020) where a meta-classifier approach is used, again the ComParE feature set is used, together with the ASR information and the spectrogram representation for the e2e approach.…”

“…Furthermore, three studies employed an e2e classification path, directly working on the acoustic representation (Akhtiamov et al, 2019(Akhtiamov et al, , 2020.…”