Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

Lücking, Andy; Bergman, Kirsten; Hahn, Florian; Kopp, Stefan; Rieser, Hannes

doi:10.1007/s12193-012-0106-8

Cited by 39 publications

(27 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Previous works have investigated various aspects of the production, perception, processing, and development of gestures (e.g., Kelly & Church 1998;Kelly & Barr 1999;Mayberry & Nicoladis 2000;McNeil et al 2000;O'Neill et al 2002;Holler & Beattie 2003a;b;Holle & Gunter 2007;Özyürek et al 2007;Alibali et al 2009;Gullberg 2009;Kelly et al 2009;Kidd & Holler 2009;Botting et al 2010;Göksun et al 2010;Cartmill et al 2012;Dick et al 2012;Lücking et al 2012;Özçalişkan & Dimitrova 2013;Emmorey & Özyürek 2014;Hrabic et al 2014;Özyürek 2014;Wagner et al 2014). While many of these existing studies have examined the meanings that co-speech gestures contribute, they do not target the precise ways in which gestures may interact with the logical structure of the sentences with which they co-occur.…”

Section: Experimental Investigationmentioning

confidence: 99%

Co-speech gesture projection: Evidence from truth-value judgment and picture selection tasks

Tieu

Pasternak

Schlenker

et al. 2017

Glossa: A Journal of General Linguistics

View full text Add to dashboard Cite

Two main analyses have been proposed to explain how co-speech gestures interact with logical operators. According to the Supplemental analysis (Ebert & Ebert 2014), co-speech gestures have the same semantic status as appositive relative clauses. According to the Cosuppositional analysis (Schlenker To appear a; b), co-speech gestures trigger a particular kind of presupposition. The sentence "John will not use the stairs", produced with an UP gesture (finger pointed upwards) is argued to give rise to the conditional presupposition that if John were to use the stairs, he would use the stairs in an upwards trajectory. Both the Supplemental and Cosuppositional analyses predict that inferences triggered by co-speech gestures should project out of the scope of operators, but not quite in the same way. We present an experimental investigation of the projection properties of the inferences arising from the co-speech gestures UP and DOWN in six different linguistic environments (plain affirmative and negative sentences, modal sentences containing "might", and quantified sentences containing "each", "none", and "exactly one"). Applying a reading detection analysis (Cremers & Chemla 2017) to the responses of a Truth Value Judgment Task and a Picture Selection Task, we find evidence for existential projection of the gestural inferences in the scope of "each", "none", and "exactly one", and, to some degree, local accommodation of the inferences. These results can be derived by the Cosuppositional analysis, in combination with an analysis of presupposition projection such as Beaver (2001), which predicts existential projection out of quantified structures; on the other hand, both findings are difficult to reconcile with the Supplemental analysis. Our projection results bring gestural inferences and verbal presuppositions closer together, but a remaining puzzle is why in quantified structures we obtain existential rather than universal inferences (Chemla 2009), the latter being the more standard finding in the presuppositional literature (though see Tieu et al. 2016 for evidence of universal projection of the same gestural inferences).

show abstract

Section: Experimental Investigationmentioning

confidence: 99%

Co-speech gesture projection: Evidence from truth-value judgment and picture selection tasks

Tieu

Pasternak

Schlenker

et al. 2017

Glossa: A Journal of General Linguistics

View full text Add to dashboard Cite

show abstract

“…An example analysis of the internal structure of an instance of creative gesticulation is given in Figure 5. The analysis concerns a fragment of the Speech and Gesture Alignment (SaGA) corpus (Lücking et al 2013), which contains video recordings of German direction giving discourse. As seen in Figure 4, speech and gesture are employed in tandem to describe a physical landmark that is relevant for the route description.…”

Section: Gesture-internal Compositionality: Conventional and Ad Hoc Amentioning

confidence: 99%

Cognitive Grammar and gesture: Points of convergence, advances and challenges

Kok

Cienki²

2016

Cognitive Linguistics

View full text Add to dashboard Cite

Given its usage-oriented character, Cognitive Grammar (CG) can be expected to be consonant with a multimodal, rather than text-only, perspective on language. Whereas several scholars have acknowledged this potential, the question as to how speakers' gestures can be incorporated in CG-based grammatical analysis has not been conclusively addressed. In this paper, we aim to advance the CG-gesture relationship. We first elaborate on three important points of convergence between CG and gesture research: (1) CG's conception of grammar as a prototype category, with central and more peripheral structures, aligns with the variable degrees to which speakers' gestures are conventionalized in human communication. (2) Conceptualization, which lies at the basis of grammatical organization according to CG, is known to be of central importance for gestural expression. In fact, all of the main dimensions of construal postulated in CG (specificity, perspective, profile-base relationship, conceptual archetypes) receive potential gestural expression. (3) CG's intensive use of diagrammatic notation allows for the incorporation of spatial features of gestures. Subsequently, we demonstrate how CG can be applied to analyze the structure of multimodal, spoken-gestured utterances. These analyses suggest that the constructs and tools developed by CG can be employed to analyze the compositionality that exists within a single gesture (between conventional and more idiosyncratic components) as well as in the grammatical relations that may exist between gesture and speech. Finally, we raise a number of theoretical and empirical challenges.

show abstract

“…A informação da trajetória das mãos e da posição de partes do corpoé explorada por Abid et al [1], na tarefa de reconhecimento e classificação de gestos em tempo real, e por Liu et al [17], no reconhecimento de atividades humanas complexas. A trajetória junto com o posicionamento de objetos com o qual as pessoas estão interagindo nas cenas capturadas compõem a representação de dados usada por Lücking et al [21] e Rosani et al [27] também na tarefa de reconhecimento e classificação de gestos em tempo real.…”

Section: Formas De Representação De Dadosunclassified

“…Modelo probabilístico e Grafos AND/OR Hierarquia na representação das poses [3] 95 Segmental HMM Estados e transições; predição estruturada [17] 93,4 Markov Semantic Model Estados e transições; predição estruturada [24] 91,76 HMM, SVM Estados e transições; predição estruturada [14] -Inspeção visual (humana) Analisa características linguísticas [21] -Gramáticas Uso de gramáticas para tomada de decisão [4] 100 Rede neural artificial recorrente Análise de sequências (temporais) [7] 96,23 K-means, SVM e Gramáticas Uso de gramáticas para tomada de decisão [9] 98,5…”

Section: Estudo Acuráciaunclassified

Representação de Gestos em Análise Sintática: uma Revisão Sistemática da Literatura

Feitosa

Peres

Lima

2016

Anais Do Simpósio Brasileiro De Sistemas De Informação (SBSI)

View full text Add to dashboard Cite

RESUMOSistemas de análise de gestos vêm ganhando força por sua capacidade de contribuir com a interação entre humanos, humanos e máquinas, e humanos e ambiente. Nesses sistemas, o estabelecimento de uma representação de dados eficiente para os gestosé uma tarefa crítica. A representação escolhida e sua associação a técnicas de análise podem ou não favorecer a solução sob implementação. Nesta Revisão Sistemática são identificadas e discutidas as estratégias de representação de gestos usadas em 21 estudos publicados noś ultimos 5 anos, no contexto da análise sintática de gestos. ABSTRACTGestures analysis systems have been getting attention for their ability to contribute to the interaction between humans, humans and machines, and humans and environments. In such systems, the establishment of an efficient data representation for gestures is a critical task. The chosen representation as well as its combination with techniques for analysis can or can not favor the solution being developed. In this systematic review we identify and discuss the strategies of representation of gestures used in 21 studies published in the last five years in the context of syntactic gesture analysis.

show abstract

Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

Cited by 39 publications

References 40 publications

Co-speech gesture projection: Evidence from truth-value judgment and picture selection tasks

Co-speech gesture projection: Evidence from truth-value judgment and picture selection tasks

Cognitive Grammar and gesture: Points of convergence, advances and challenges

Representação de Gestos em Análise Sintática: uma Revisão Sistemática da Literatura

Contact Info

Product

Resources

About