Using affective brain-computer interfaces to characterize human influential factors for speech quality-of-experience perception modelling

Abstract

As new speech technologies emerge, telecommunication service providers have to provide superior user experience in order to remain competitive. To this end, quality-of-experience (QoE) perception modelling and measurement has become a key priority. QoE models rely on three influence factors: technological, contextual and human. Existing solutions have typically relied on the former two and human influence factors (HIFs) have been mostly neglected due to difficulty in measuring them. In this paper, we show that measuring human affective states is important for QoE measurement and propose the use of affective brain-computer interfaces (aBCIs) for objective measurement of perceived QoE for two emerging speech technologies, namely far-field hands-free communications and text-to-speech systems. When incorporating subjectively-derived HIFs into the QoE model, gains of up to 26.3 % could be found relative to utilizing only technological factors. When utilizing HIFs derived from an electroencephalography (EEG) based aBCI, in turn, gains of up to 14.5 % were observed. These findings show the importance of using aBCIs in QoE measurement and also highlight that further improvement may be warranted once improved affective state correlates are found from EEGs and/or other neurophysiological modalities.

Multimodal Physiological Quality-of-Experience Assessment of Text-to-Speech Systems

“…Recently, HIFs and objective HIF characterization have gained burgeoning attention from QoE researchers. Towards combining HIFs, such as affective states, with technologycentric speech quality metrics, researchers have investigated the use of EEG in [4]. Here, we investigated the use of fusion of physiological modalities to model affective states.…”

“…Next, the pre-processed EEG, fNIRS and HRV time series signals were then used to extract features that encode users' valence and arousal thus, forming physiological feature set. As such, EEG signals were used to extract graph theoretical features, such as local efficiency (E l ) and global efficiency (E g ), as described in [6]; and asymmetry index and medial beta power (MBP), as described in [4]. Also, using fNIRS signals, certain features, such as average and peak values for ∆[HbO] and ∆[HbR] concentrations, as described in [7] were extracted.…”

“…Moreover, leveraging HRV time series statistical features, such as mean, median, skewness and kurtosis, were extracted. Finally, two quantitative measures were extracted from speech stimuli themselves to model the technological influence factors, as described in [4]. The measures included the slope of the second order derivative of the fundamental frequency (sF 0 ) and the absolute mean of the second order mel frequency cepstrum coefficient (M F CC 2 ).…”

“…The recent advances in wireless and portable neurotechnologies have lead to the development of a plethora of applications, such as to measure implicit information from the users including their mental states (e.g., stress level), fatigue levels, and more recently, their mood and emotional states [3]. In fact, EEG has proved effective in modelling HIFs, such as affective states [4]. In this paper, we have compared the performances of fusion of multiple physiological modalities and individual modalities for monitoring the human influential factors needed in Quality-of-Experience (QoE) perception models.…”

Speech Quality Evaluation of Synthesized Japanese Speech Using EEG

“…A previous study showed that electroencephalographs (EEG) could reveal the correlation between pitch emphasis and brain activity [7]. In [8], they proposed brain computer interface-based equation to predict quality of experience MOS, and achieved 1.00 of root mean squared error (RMSE) between actual and predicted MOS. In addition, by using tensor representation of all channels and all frequency bands, a study conducted by [9] shows that EEG signals could be used to predict MOS, valence, and arousal within the same subject.…”