Comparing one and two-stage acoustic modeling in the recognition of emotion in speech

Schuller, Bjoern; Vlasenko, Bogdan; Mínguez, Rikardo; Rigoll, Gerhard; Wendemuth, Andreas

doi:10.1109/asru.2007.4430180

Cited by 30 publications

(16 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Taking into account current affective cues, responses have to be prepared already before the user has finished speaking and hypotheses have to be generated onthe-fly. It might not make sense at first to detect emotion from speech every few milliseconds, since emotion remains bound to syllables or even words [31]. However, if the output is smoothed over short time periods, reliable estimates of quasi-instantaneous emotions can be given, without the need to identify word boundaries, for example.…”

Section: Introductionmentioning

confidence: 99%

On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues

Eyben

Wöllmer

Graves

et al. 2009

J Multimodal User Interfaces

128

View full text Add to dashboard Cite

For many applications of emotion recognition, such as virtual agents, the system must select responses while the user is speaking. This requires reliable on-line recognition of the user's affect. However most emotion recognition systems are based on turnwise processing. We present a novel approach to on-line emotion recognition from speech using Long Short-Term Memory Recurrent Neural Networks. Emotion is recognised frame-wise in a two-dimensional valence-activation continuum. In contrast to current state-of-the-art approaches, recognition is performed on low-level signal frames, similar to those used for speech recognition. No statistical functionals are applied to low-level feature contours. Framing at a higher level is therefore unnecessary and regression outputs can be produced in real-time for every low-level input frame. We also investigate the benefits of including linguistic features on the signal frame level obtained by a keyword spotter.

show abstract

Section: Introductionmentioning

confidence: 99%

On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues

Eyben

Wöllmer

Graves

et al. 2009

J Multimodal User Interfaces

128

View full text Add to dashboard Cite

show abstract

“…The term 'segment' will be used in the ongoing referring to a general unit of analysis. Finding the optimal unit of analysis is still an active area of research Schuller et al 2007b;Schuller et al 2008;Busso et al 2007;. As stated in [Zeng et al 2009], the segmentation is one of the most important issues for real applications but has been "largely unexplored so far".…”

Section: The Unit Of Analysismentioning

confidence: 99%

A multitask approach to continuous five-dimensional affect sensing in natural speech

Eyben¹,

Wöllmer²,

Schuller³

2012

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

Automatic affect recognition is important for the ability of future technical system to interact with us socially in an intelligent way by understanding our current affective state. In recent years there has been a shift in the field of affect recognition from "in the lab" experiments with acted data to "in the wild" experiments with spontaneous and naturalistic data. Two major issues thereby are the proper segmentation of the input and adequate description and modelling of affective states. The first issue is crucial for responsive, real-time systems such as virtual agents and robots, where the latency of the analysis must be as small as possible. To address this issue we introduce a novel method of incremental segmentation to be used in combination with supra-segmental modelling For modelling of continuous affective states we use Long Short-Term Memory Recurrent Neural Networks, with which we can show an improvement in performance over standard recurrent neural networks and feed forward neural networks as well as Support Vector Regression. For experiments we use the SEMAINE database, which contains recordings of spontaneous and natural human to Wizard-of-Oz conversations. The recordings are annotated continuously in time and magnitude with FeelTrace for five affective dimensions, namely activation, expectation, intensity, power/dominance, and valence. To exploit dependencies between the five affective dimensions we investigate multi-task learning of all five dimensions augmented with inter-rater standard deviation. We can show improvements for multi-task over single task modelling. Correlation coefficients of up to 0.81 are obtained for the activation dimension and up to 0.58 for the valence dimension. The performance for the remaining dimensions were found to be in between that for activation and valence.

show abstract

“…Speech and speaker recognition techniques: short-term features and statistical modeling (GMM, HMM) have been successfully combined with a traditional turn based level approach [15]. In [16], a timescale is identified by a the extraction of short-term feature extraction (25 ms windows, MFCC) and the use of statistical modeling (HMM). The time-scale is called by the authors chunk level.…”

Section: Machine Learning Based Unitsmentioning

confidence: 99%

“…''Combining framelevel and segment-based Approach for Intention Recognition in infant-directed Speech'') Section : the idea is to exploit speaker recognition techniques which are mainly based on frame-level modeling (all the frames are exploited for the characterization) as it is done in [16,18]. -data-driven approach (see ''Data-Driven Approach for Time-Scale Feature Extraction'') : speech signals are characterized by prominent segments such as vowels which are then employed as sub-units.…”

Section: Data-fusion Approachmentioning

confidence: 99%

“…For instance, voiced segments are known to convey more relevant information about emotion and focusing on these segments has been proven to be efficient [1,12]. Various methods have been investigated for combining different levels [12,[14][15][16][17]. In [12], the Segment Based Approach (SBA) proposes to divide the whole utterance (turn level) on N voiced segments and then to characterize each voiced segments.…”

Section: Data-driven Unitsmentioning

confidence: 99%

See 1 more Smart Citation

Time-Scale Feature Extractions for Emotional Speech Characterization

2009

View full text Add to dashboard Cite

Emotional speech characterization is an important issue for the understanding of interaction. This article discusses the time-scale analysis problem in feature extraction for emotional speech processing. We describe a computational framework for combining segmental and supra-segmental features for emotional speech detection. The statistical fusion is based on the estimation of local a posteriori class probabilities and the overall decision employs weighting factors directly related to the duration of the individual speech segments. This strategy is applied to a real-world application: detection of Italian motherese in authentic and longitudinal parent-infant interaction at home. The results suggest that short-and long-term information, respectively, represented by the short-term spectrum and the prosody parameters (fundamental frequency and energy) provide a robust and efficient time-scale analysis. A similar fusion methodology is also investigated by the use of a phonetic-specific characterization process. This strategy is motivated by the fact that there are variations across emotional states at the phoneme level. A timescale based on both vowels and consonants is proposed and it provides a relevant and discriminant feature space for acted emotion recognition. The experimental results on two different databases Berlin (German) and Aholab (Basque) show that the best performance are obtained by our phoneme-dependent approach. These findings demonstrate the relevance of taking into account phoneme dependency (vowels/consonants) for emotional speech characterization.

show abstract

Comparing one and two-stage acoustic modeling in the recognition of emotion in speech

Cited by 30 publications

References 3 publications

On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues

On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues

A multitask approach to continuous five-dimensional affect sensing in natural speech

Time-Scale Feature Extractions for Emotional Speech Characterization

Contact Info

Product

Resources

About