Speaking and gesturing guide event perception during message conceptualization: Evidence from eye movements

Multimodal encoding of motion events in speech, gesture and cognition

“…On the other hand, speakers of Turkish and Japanese typically distributed path and manner information across different clauses in speech and also tended to produce separate gestures for path and manner. Similar findings have been replicated across different languages and language pairs, such as English (Kita et al, 2007), Farsi (Akhavan et al, 2017), French (Gullberg et al, 2008), Turkish (Mamus et al, 2022(Mamus et al, , 2023Ünal et al, 2022), Dutch-Turkish (ter Bekke et al, 2022), Turkish-English (Özçalışkan et al, 2016a, 2016b; Özyürek et al, 2005), Korean-English (Choi & Lantolf, 2008) and Japanese-English-Turkish (Kita & Özyürek, 2003). Thus, co-speech gesture often follows the typological patterns in motion event encoding defined by Talmy (2000).…”

“…Most studies reviewed so far have used visual stimuli to examine motion event expressions and their patterns by using video-clips, cartoons, line drawings and so on (Akhavan et al, 2017;Gennari et al, 2002;Gullberg et al, 2008;Kita & Özyürek, 2003;Papafragou et al, 2002;Slobin et al, 2014;ter Bekke et al, 2022;Ünal et al, 2022). These studies have not taken into account whether these patterns might change depending on the modality of the input.…”

“…Thus, gestures encode additional information to speech when this information is difficult to encode linguistically. However, later empirical work provided abundant evidence against the claims of McNeill and Duncan (2000), and showed that speech and gesture typically express similar content as co-speech gestures appear with the element expressed in the main verb (Akhavan et al, 2017;Gullberg et al, 2008;Kita & Özyürek, 2003;Özyürek et al, 2005;ter Bekke et al, 2022;Ünal et al, 2022). For example, when English and Turkish speakers encode both path and manner, they package them in syntactically different ways both in speech and in co-speech gesturethat is, English speakers producing conflated gestures more frequently and Turkish speakers producing separate path or manner gestures more frequently.…”

Human–Machine Interaction via Dual Modes of Voice and Gesture Enabled by Triboelectric Nanogenerator and Machine Learning

“…Nowadays, the rapid development of artificial intelligence and the Internet of Things (IoT) has brought great changes to people’s lives, and human–machine interaction sets application requirements and research ideas for artificial intelligence and the Internet of Things (IoT). − The technology of how to improve the efficiency of communication between human and machine is the most difficult problem for the researchers to overcome, in which the communication system − connecting human and machine is particularly important. Nowadays, there are two main methods of human–machine interaction: voice–machine interaction − and gesture–machine interaction. − Voice signals contain rich character information on humans and are easy to use, so people can use voice signals to make and realize accurate commands. At present, the main sensors currently used for speech recognition are capacitive sensors, , piezoresistive sensors, , piezoelectric sensors, − and triboelectric sensors. − However, some components in capacitive and piezoresistive sensors, such as conventional commercial microphones, rely on battery charging and are vulnerable to environmental noise.…”