Toward emotion indexing of multimedia excerpts

Abstract

Multimedia indexing is about developing techniques allowing people to effectively find media. Content-based methods become necessary when dealing with large databases. Current technology allows exploring the emotional space which is known to carry very interesting semantic information. In this paper we state the need for an integrated method which extracts reliable affective information and attaches this semantic information to the medium itself. We describe SAMMI [1], a framework explicitly designed to fulfill this need and we present a list of possible applications pointing out the advantages that the emotional information can bring about. Finally, different scenarios are considered for the recognition of the emotions which involve different modalities, feature sets, fusion algorithms, and result optimization methods such as temporal averaging or thresholding.

Towards Efficient Multi-Modal Emotion Recognition

“…Additionally, we study the applicability of several fusion schemes to further improve upon the results obtained with the individual modalities. Our results show that both unimodal approaches as well as the proposed combined system compare favorably with state-of-the-art techniques from the literature [9,12,13,14].…”

“…. Confusion matrices for all 5 folds of our cross validation procedure generated using the presented audio sub-system Video SAMMI framework [12,24] 28.0 / Video sub-system [13] 37.0 / LBPs+HMMs [14] 37 …”

“…Audio Video HMM [14] 56.3 / SAMMI framework [12,24] 67.0 / Async. feature fusion [13] 71 The differences between the weighted sum-rule and weighted product-rule are minor, with the highest UW recall of 77.5% achieved by the weighted product-rule fusion procedure.…”

What Are You Trying to Say? Format-Independent Semantic-Aware Streaming and Delivery

“…The subtle exchange of glances between Elizabeth and her father would be readily apparent to most human observers, but it is unlikely that a computer processing a video of the scene would be able to recognise their meaning. Furthermore, while the double-entendre in Mr Bennett's remark would be clear to most human listeners, algorithmic recoginition of this or other modes of speech are in their infancy (Paleari & Huet, 2008). Other research communities are developing means to communicate such semantic information (whether computed or manually generated) in ways that are able to transcend the original context of the information.This work-originating from Knowledge Representation, but more popularly known as the Semantic Web-has provided languages such as the Resource Description Framework (RDF) (Beckett, 2004) and Web Ontology Language (OWL) (Dean & Schreiber, 2004) which can be used to express concepts in such a way that "this picture has many buildings" may also imply that "it is a cityscape", and "it contains man-made objects."…”

Towards an intelligent framework for multimodal affective data analysis

“…Paleari et al [69] carried out both decision and feature-level fusion. They experimented with the eNTERFACE dataset and showed that decision-level fusion outperformed feature-level fusion.…”