The social signal interpretation (SSI) framework

Wagner, Johannes; Lingenfelser, Florian; Baur, Tobias; Damian, Ionuț; Kistler, Felix; André, Elisabeth

doi:10.1145/2502081.2502223

Cited by 138 publications

(17 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interpretation & NovA. The Social Signal Interpretation (SSI) [37] is an open source framework for real-time recognition of social signals during social interactions. SSI supports synchronised data recording from a large range of sensor devices to recognise behavioural cues such as gestures, head movements, and emotional speech.…”

Section: Social Signalmentioning

confidence: 99%

Read Between the Lines

Mitri

Schneider

Klemke

et al. 2019

Proceedings of the 9th International Conference on Learning Analytics &Amp; Knowledge

View full text Add to dashboard Cite

This paper introduces the Visual Inspection Tool (VIT) which supports researchers in the annotation of multimodal data as well as the processing and exploitation for learning purposes. While most of the existing Multimodal Learning Analytics (MMLA) solutions are tailor-made for specific learning tasks and sensors, the VIT addresses the data annotation for different types of learning tasks that can be captured with a customisable set of sensors in a flexible way. The VIT supports MMLA researchers in 1) triangulating multimodal data with video recordings; 2) segmenting the multimodal data into time-intervals and adding annotations to the time-intervals; 3) downloading the annotated dataset and using it for multimodal data analysis. The VIT is a crucial component that was so far missing in the available tools for MMLA research. By filling this gap we also identified an integrated workflow that characterises current MMLA research. We call this workflow the Multimodal Learning Analytics Pipeline, a toolkit for orchestration, the use and application of various MMLA tools.

show abstract

Section: Social Signalmentioning

confidence: 99%

Read Between the Lines

Mitri

Schneider

Klemke

et al. 2019

Proceedings of the 9th International Conference on Learning Analytics &Amp; Knowledge

View full text Add to dashboard Cite

show abstract

“…The architecture of this interface is highly dependent on whether a fusion method is integrated into a RIS platform or stand alone. The latter include frameworks like OpenInterface [54], Mudra [40], or SSI [55]. They provide functionality for integrating sensors and performing fusion with the goal to foster modularity and reusability.…”

Section: Contextmentioning

confidence: 99%

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

Zimmerer

Fischbach

Latoschik

2018

MTI

View full text Add to dashboard Cite

Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept’s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality.

show abstract

“…Sensing and processing the user's social cues is an essential part of the presented adaptation process and prototype. A Microsoft Kinect 2 sensor is used to capture the human user's movements in combination with the SSI framework [30] which processes and interprets the data in real-time. To calculate ∆E t as reward signal for RL, we interpret the user's behaviors and derive the current engagement value E t .…”

Section: B Estimating User Engagementmentioning

confidence: 99%

Adapting a Robot's linguistic style based on socially-aware reinforcement learning

Ritschel

Baur

André

2017

2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)

Self Cite

View full text Add to dashboard Cite

When looking at Socially Interactive Robots, adaptation to the user's preferences plays an important role in today's Human-Robot Interaction to keep interaction interesting and engaging over a long period of time. Findings indicate an increase in user engagement for robots with adaptive behavior and personality, but also that it depends on the task context whether a similar or opposing robot personality is preferred. We present an approach based on Reinforcement Learning, which gets its reward directly from social signals in real-time during the interaction, to quickly learn about and dynamically address individual human preferences. Our scenario involves a Reeti robot in the role of a story teller talking about the main characters in the novel "Alice's Adventures in Wonderland" by generating descriptions with varying degree of introversion/extraversion. After initial simulation results, an interactive prototype is presented which allows to explore the learning process adapting to the human interaction partner's engagement.Paper accepted for 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

show abstract

The social signal interpretation (SSI) framework

Cited by 138 publications

References 13 publications

Read Between the Lines

Read Between the Lines

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

Adapting a Robot's linguistic style based on socially-aware reinforcement learning

Contact Info

Product

Resources

About