Multimodal Emotion Analysis Based on Acoustic and Linguistic Features of the Voice

Koren, Leon; Stipančić, Tomislav

doi:10.1007/978-3-030-77626-8_20

Cited by 6 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to the model developed and presented in [3], this work will continue by exploring how the robot can hear all people in vicinity but "concentrate" on only one.…”

Section: Discussionmentioning

confidence: 99%

“…The model is implemented for use on the interactive biomimetic robotic head PLEA [2]. Based on the multimodal approach, PLEA uses visual and voice modalities fused in a separate algorithm to determine the most appropriate hypothesis about the person's emotional state at a given moment [3]. These decisions are changeable over time as the system receives the latest information.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

et al. 2022

Self Cite

View full text Add to dashboard Cite

PLEA is an interactive, biomimetic robotic head with non-verbal communication capabilities. PLEA reasoning is based on a multimodal approach combining video and audio inputs to determine the current emotional state of a person. PLEA expresses emotions using facial expressions generated in real-time, which are projected onto a 3D face surface. In this paper, a more sophisticated computation mechanism is developed and evaluated. The model for audio-visual person separation can locate a talking person in a crowded place by combining input from the ResNet network with input from a hand-crafted algorithm. The first input is used to find human faces in the room, and the second input is used to determine the direction of the sound and to focus attention on a single person. After an information fusion procedure is performed, the face of the person speaking is matched with the corresponding sound direction. As a result of this procedure, the robot could start an interaction with the person based on non-verbal signals. The model was tested and evaluated under laboratory conditions by interaction with users. The results suggest that the methodology can be used efficiently to focus a robot’s attention on a localized person.

show abstract

“…In addition to the model developed and presented in [3], this work will continue by exploring how the robot can hear all people in vicinity but "concentrate" on only one.…”

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…By using the entropy reduction method, it is possible to determine a drop in uncertainty in query nodes when evidence or proof is provided to some particular node in the network, as shown in Eq. (11).…”

Section: Figure 4 Ambiguities In Bn Reasoningmentioning

confidence: 99%

“…The partial information currently concluded or acquired by senses can instantly change the reasoning output. In this way, a single piece of information can result in recognition, or it can result in a change in perspective [11].…”

Section: Introductionmentioning

confidence: 99%

Context-Driven Method in Realization of Optimized Human-Robot Interaction

Koren

Stipančić

Ričko

et al. 2022

Teh. glas. (Online)

Self Cite

View full text Add to dashboard Cite

Perceptual uncertainty and environmental volatility are among the most enduring challenges in robotic research today. Contemporary robotic systems are usually designed to work in specific and controlled domains where a total number of variables is defined. Traditional solutions therefore often result in over-constrained interaction spaces or rigid system architectures where any unexpected change can result in system failure. The focus of this work is set on achieving a constant adaptation of the system to changes through interaction. A computational mechanism based on the entropy reduction method is integrated along with the three-component control model. This model is seen as a context-to-data interpreter used to provide context-aware reasoning to the technical system. The mechanism is using a decrease in interaction uncertainties when proofs are provided to the system. In this way, the robot can choose the right interaction strategy that resolves reasoning ambiguities most efficiently

show abstract