Multimodal corpus of multiparty conversations in L1 and L2 languages and findings obtained from it

Abstract

To investigate the differences in communicative activities by the same interlocutors in Japanese (their L1) and in English (their L2), an 8-h multimodal corpus of multiparty conversations was collected. Three subjects participated in each conversational group, and they had conversations on free-flowing and goaloriented topics in Japanese and in English. Their utterances, eye gazes, and gestures were recorded with microphones, eye trackers, and video cameras. The utterances and eye gazes were manually annotated. Their utterances were transcribed, and the transcriptions of each participant were aligned with those of the others along the time axis. Quantitative analyses were made to compare the communicative activities caused by the differences in conversational languages, the conversation types, and the levels of language expertise in L2. The results reveal different utterance characteristics and gaze patterns that reflect the differences in difficulty felt by the participants in each conversational condition. Both total and average durations of utterances were shorter in their L2 than in their L1 conversations. Differences in eye gazes were mainly found in those toward the information senders: Speakers were gazed at more in their second-language than in their native-language conversations. Our findings on the characteristics of conversations in the second language suggest possible directions for future research in psychology, cognitive science, and humancomputer interaction technologies.

Comparison of Effect of Speaker's Eye Gaze on Selection of Next Speaker between Native- and Second-Language Conversations

“…A multimodal corpus created by Yamamoto et al [17] was used for comparing the effect of eye gaze on selection of the next speaker between native-and second-language conversations. Three subjects participated in a conversational group, sitting in a triangular formation around a table as shown in Figure 1.…”

“…They argued that this was because video transmitting facial information and gestures helped the non-native pairs to negotiate a common ground, whereas this did not provide significant help for the native pairs. These observations suggest that eye gaze and visual information play more important roles in establishing mutual understanding in L2 conversations than To quantitatively and precisely analyze the difference in eye gaze between L1 and L2 conversations, Yamamoto et al [17] created a multimodal corpus of three-party conversations for two different conversation topics in L1 and L2. In this way, it was possible to compare the features of utterance, eye gaze, and body posture in L1 and L2 conversations conducted by the same interlocutors [17].…”

“…These observations suggest that eye gaze and visual information play more important roles in establishing mutual understanding in L2 conversations than To quantitatively and precisely analyze the difference in eye gaze between L1 and L2 conversations, Yamamoto et al [17] created a multimodal corpus of three-party conversations for two different conversation topics in L1 and L2. In this way, it was possible to compare the features of utterance, eye gaze, and body posture in L1 and L2 conversations conducted by the same interlocutors [17]. To compare the features of eye gaze in L1 and L2 conversations, they used two metrics: (1) how long the speaker was gazed at by other participants during her or his utterance (listener's gazing ratio) and (2) how long the speaker gazed at other participants during her or his utterance (speaker's gazing ratio).…”

Difference in Eye Gaze for Floor Apportionment in Native- and Second-Language Conversations

“…To quantitatively and precisely analyze the difference in eye gaze between L1 and L2 conversations, Yamamoto et al ( 2015 ) created a multimodal corpus of three-party conversations for two different conversation topics in L1 and L2. In this way, it was possible to compare the features of utterance, eye gaze, and body posture in L1 and L2 conversations conducted by the same interlocutors (Yamamoto et al 2015 ).…”

“…To quantitatively and precisely analyze the difference in eye gaze between L1 and L2 conversations, Yamamoto et al ( 2015 ) created a multimodal corpus of three-party conversations for two different conversation topics in L1 and L2. In this way, it was possible to compare the features of utterance, eye gaze, and body posture in L1 and L2 conversations conducted by the same interlocutors (Yamamoto et al 2015 ). To compare the features of eye gaze in L1 and L2 conversations, they used two metrics: (1) how long the speaker was gazed at by other participants during her or his utterance (listener’s gazing ratio) and (2) how long the speaker gazed at other participants during her or his utterance (speaker’s gazing ratio).…”