Hearing and Seeing Tone Through Color: An Efficacy Study of Web‐Based, Multimodal Chinese Tone Perception Training

Abstract

Multimodal approaches have been shown to be effective for many learning tasks. In this study, we compared the effectiveness of five multimodal methods for second language (L2) Mandarin tone perception training: three single‐cue methods (number, pitch contour, color) and two dual‐cue methods (color and number, color and pitch contour). A total of 303 true novice learners of L2 Mandarin (native speakers of English) completed a 3‐week online training program. Results from pretests as well as immediate and delayed posttests indicated that multimodal training aided L2 learners’ tone perception, with a small, practical advantage for pitch contours and numbers over color coding. Dual‐cue methods did not yield better learning than single‐cue training. Thus, the additive benefits of multimodal input (i.e., auditory and visual) did not extend to instruction featuring doubled visual input (i.e., visual and visual). We argue for embedding color in visuals in a way that helps make abstract information concrete.

The phonetics of second language learning and bilingualism

“…Either or both of these factors may be responsible for the observed interference associated with varying task-irrelevant phonetic features in L2 speech training (Antoniou and Wong, 2016). Godfroid et al (2017), however, are careful to point out that performance in their dual-cue condition was never worse than in the single-cue condition, which is not entirely consistent with an account of their results in terms of cognitive load or extraneous processing. Consequently, they explain the lack of benefit associated with adding a second cue in terms of an implementation issue: given the way in which color was incorporated into the stimuli, participants may have perceptually backgrounded the color cue, such that "color might have played a more peripheral role than intended" (Godfroid et al, 2017, p. 846).…”

“…However, two recent studies suggest that multimodal L2 engagement does not necessarily improve L2 outcomes, and in certain cases can actually be detrimental to L2 speech development. In one study of L1 English speakers being trained on Mandarin tones (Godfroid, Lin and Ryu, 2017), several types of multimodal perceptual training were systematically compared to each other, including three "single-cue" types of training (involving exposure to only one visual cue alongside auditory stimuli: number, pitch contour, or color) and two "dual-cue" types of training (involving exposure to two visual cues: color and number, or color and pitch contour). Although test results showed perceptual gains with all training types, single-cue exposure to numbers or pitch contours was more beneficial than single-cue exposure to colors, while neither dual-cue exposure was more beneficial than single-cue exposure.…”

“…There are at least three possible, and not mutually exclusive, explanations. Baese-Berk and Samuel (2016) allude to the role of cognitive (over)load in this type of multimodal setting (see, e.g., van Merriënboer and Sweller, 2005), while Godfroid et al (2017) additionally consider the potentially detrimental effect of extraneous processing of redundant or irrelevant information. Either or both of these factors may be responsible for the observed interference associated with varying task-irrelevant phonetic features in L2 speech training (Antoniou and Wong, 2016).…”

Do Explicit Instruction and High Variability Phonetic Training Improve Nonnative Speakers’ Mandarin Tone Productions?

“…Given the challenges associated with L2 tone acquisition, there is increasing interest in exploring what potential pedagogical strategies may result in more native‐like performance (Duff & Li, ). Most recently, researchers in this area have examined the use of adaptive training systems (Shih et al., ), web‐based training platforms (Godfroid, Lin, & Ryu, ), visuospatial gestures (Morett & Chang, ), and incidental learning videogames (Wiener, Murphy, et al, ), among other approaches. We contribute to this ongoing discussion by investigating how explicit instruction of tone contours and high variability phonetic training affect the production of L2 Mandarin tones.…”

Nonlinguistic Factors that Affect the Degree of Foreign Accent in Second Language Mandarin

“…Previous research into L2 Mandarin phonology has largely focused on the perception of tone errors and the effects of having a tonal or nontonal L1 (e.g., Godfroid et al 2017). For example, in the study by So and Best (2010), L1 Cantonese and Japanese speakers (both of which represent tonal L1s) outperformed L1 English speakers when identifying L2 Mandarin tones.…”