On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA)

Rieser, Hannes

doi:10.1007/978-3-642-12553-9_5

Cited by 8 publications

(11 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, some hand shapes function as "line-thickness modifiers" of the gesture trajectory, giving rise to a three-dimensional body rather than to a two-dimensional sketch (cf. the work of [29]). Some rules that will be used below are collected in Table I by example of the sagittal plane.…”

Section: B Trajectories Within Vector Spacementioning

confidence: 97%

Modeling Co-Verbal Gesture Perception in Type Theory with Records

Lücking¹

2016

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

In natural language face to face communication interlocutors exploit manifold non-verbal information resources, most notably hand and arm movements, i.e. gestures. In this paper, a type-theoretical approach using Type Theory with Records is introduced which accounts for iconic gestures within an information state update semantics. Iconic gestures are semantically exploited in two steps: firstly, their kinetic representations are mapped onto vector sequence representations from vector space semantics, modeling a perceptual gesture classification; secondly, these vectorial representations are linked to linguistic predicates, giving rise to a computational account to semantic-kinematic interfaces. Each of the steps involves reasoning processes, which are made explicit. The resulting framework shows how various resources have to be integrated in the update mechanism in order to deal with apparently simple multimodal utterances.

show abstract

Section: B Trajectories Within Vector Spacementioning

confidence: 97%

Modeling Co-Verbal Gesture Perception in Type Theory with Records

Lücking¹

2016

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

show abstract

“…Recurrent gesture information was captured in the manual multi-modal annotation of the data. It was assembled in types, represented as typed feature structures and coded in AVMs (see [57]). Types are extracted manually considering which features and information packages enter larger informational structures and are used in different gesture contexts.…”

Section: Gesture Typology Work Using Sagamentioning

confidence: 99%

“…Our typological work was at first restricted to pointing and iconic gestures co-occurring with noun phrases [26,57]. In the sequel we developed classifications of gestures indicating dialogue structure [28] and presently we work on gestures related to full verb phrases [60].…”

Section: Gesture Typology Work Using Sagamentioning

confidence: 99%

See 1 more Smart Citation

Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

Lücking

Bergman

Hahn

et al. 2012

J Multimodal User Interfaces

Self Cite

View full text Add to dashboard Cite

Communicating face-to-face, interlocutors frequently produce multimodal meaning packages consisting of speech and accompanying gestures. We discuss a systematically annotated speech and gesture corpus consisting of 25 route-and-landmark-description dialogues, the Bielefeld Speech and Gesture Alignment corpus (SaGA), collected in experimental face-to-face settings. We first describe the primary and secondary data of the corpus and its reliability assessment. Then we go into some of the projects carried out using SaGA demonstrating the wide range of its usability: on the empirical side, there is work on gesture typology, individual and contextual parameters influencing gesture production and gestures' functions for dialogue structure. Speech-gesture interfaces have been established extending unification-based grammars. In addition, the development of a computational model of speech-gesture alignment and its implementation constitutes a research line we focus on.

show abstract

“…The analysis of physical gesture form is an indispensable prerequisite of any account of gesture generation. We have developed a typological grid for gestures accompanying noun phrases based on the SAGA corpus to specify and characterize the physical form of co-speech iconic gestures [10].This typology specifies a hierarchy of so-called annotation predicates including the four major gestural form features (handshape, hand orientation, position, and movement characteristics) which are widely accepted in gesture research. Our typology, however, goes beyond these features in that recurrent gesture events are classified according to dimensions which have semantic impact.…”

Section: Empirical and Theoretical Perspectivementioning

confidence: 99%

How Do Iconic Gestures Convey Visuo-Spatial Information? Bringing Together Empirical, Theoretical, and Simulation Studies

Rieser

Bergmann

Kopp

2012

Gesture and Sign Language in Human-Computer Interaction and Embodied Communication

Self Cite

View full text Add to dashboard Cite

Abstract. We investigate the question of how co-speech iconic gestures are used to convey visuo-spatial information in an interdisciplinary way, starting with a corpus-based empirical and theoretical perspective on how a typology of gesture form and a partial ontology of gesture meaning are related. Results provide the basis for a computational modeling approach that allows us to simulate the production of speaker-specific gesture forms to be realized with virtual agents. An evaluation of our simulation results and our methodology shows that the model is able to successfully approximate human gestural behavior use of iconic gestures, and moreover, that gestural behavior can improve how humans rate a virtual agent in terms of eloquence, competence, human-likeness, or likeability.

show abstract

On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA)

Cited by 8 publications

References 8 publications

Modeling Co-Verbal Gesture Perception in Type Theory with Records

Modeling Co-Verbal Gesture Perception in Type Theory with Records

Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

How Do Iconic Gestures Convey Visuo-Spatial Information? Bringing Together Empirical, Theoretical, and Simulation Studies

Contact Info

Product

Resources

About