Multimodality in webconference-based language tutoring: An ecological approach integrating eye tracking

Abstract

Drawing on existing research with a holistic stance toward multimodal meaning-making, this paper takes an analytic approach to integrating eye-tracking data to study the perception and use of multimodality by teachers and learners. To illustrate this approach, we analyse two webconference tutoring sessions from a telecollaborative project involving pre-service teachers and learners of Mandarin Chinese. The tutoring sessions were recorded and transcribed multimodally, and our analysis of two types of conversational side sequences shows that the integration of eye-tracking data into an ecological approach provides richer results. Specifically, our proposed approach provided a window on the participants’ cognitive management of graphic and visual affordances during interaction and uncovered episodes of joint attention.

A multimodal corpus to study videoconference interactions for techno-pedagogical competence in second language acquisition and teacher education

“…This will enable us to understand which characteristics are specific to each language and how they impact DVC pedagogical interaction. Finally, in order incorporate eye-tracking technology into our study of TSP competence, we have implemented the ecological approach developed from small case studies on the topic (Cappellini & Hsu 2022). Studies are currently being published and are available for consultation.…”

“…The VAPVISIO project aims to fill these gaps by directly observing, through the use of eye-tracking technology and multimodal conversation analysis (Cappellini 2021), the ways in which modes are selected during DVC-based pedagogical interaction. This approach allows us to study the unfolding of tutors' TSP competence during interaction itself, offering compelling insight into the interlocutors' cognition, for instance in terms of joint attention (Cappellini & Hsu 2022, Shi & Stickler 2022.…”

“…Our multilingual and multimodal corpus was constructed in order to study the development of TSP competence by tutors and learners interacting in different DVC environments. One innovative aspect of our corpus is that it includes eye-tracking data that are integrated directly into the multimodal annotations, providing a window into how the multimodality of the DVC environment is perceived and used during the interaction itself (Cappellini & Hsu 2022). We hope that this corpus will be used to explore new research questions beyond the ones that it was originally designed to answer.…”

“…Due to lack of space, in this article we will not be able to expand on the semiotic dimension of DVC CMC. The interested reader can refer to Cappellini & Hsu (2022) for a thorough methodological discussion based on the VAPVISIO corpus.…”