A methodological synthesis and meta-analysis of judgment tasks in second language research

Abstract

Judgment tasks (JTs, often called acceptability or grammaticality judgment tasks) are found extensively throughout the history of second language (L2) research (Chaudron, 1983). Data from such instruments have been used to investigate a range of hypotheses and phenomena, from generativist theories to instructional effectiveness. Though popular and convenient, JTs have engendered considerable controversy, with concerns often centered on their construct validity in terms of the type of representations they elicit, such as implicit or explicit knowledge (Ellis, 2005; Vafaee et al., 2016). A number of studies have also examined the impact of JT conditions such as timed vs. untimed, oral vs. written (e.g., Murphy, 1997; Spada et al., 2015). This paper presents a synthesis of the use of JTs and a meta-analysis of the effects of task conditions on learner performance. Following a comprehensive search, 385 JTs were found in 302 individual studies. Each report was coded for features related to study design as well as methodological, procedural, and psychometric properties of the JTs. These data were synthesized in order to understand how this type of instrument has been implemented and reported. In addition to observing a steady increase in the use of JTs over the last four decades, we also found many of the features of JTs, when reported, varied substantially across studies. In terms of the impact of JT design, whereas modality was not found to have a strong or stable effect on learner performance (median d=.14; IQR=1.04), scores on untimed JTs tended to be substantially higher than when timed (d=1.35; IQR=1.74). In examining these features and their links to findings, this paper builds on a growing body of methodological syntheses of L2 research instrumentation (e.g., Derrick, 2016; Marsden et al., in press) and makes a number of empirically grounded recommendations for future studies involving JTs.

The use of course grades as metrics in L2 research: A systematic review

“…In'nami and Koizumi (), for example, found that listening test scores based on multiple‐choice items were consistently higher than on open‐ended items (Hedges’ g = 1.11). Comparably strong item format effects were also observed in a recent synthesis of the use of grammaticality judgment tests, which found that scores on untimed tests were substantially higher than on timed tests ( d = 1.16; Plonsky, Marsden, Crowther, Gass, & Spinner, under review). Although these and other analyses of classroom assessments have provided suggestions for improving classroom‐based assessment for both teaching and research, developing valid and reliable assessments for classroom purposes remains one of the primary challenges facing practicing L2 instructors and researchers as well (e.g., Carlsen, ; Ginther & Yan, ).…”

Effects of Second Language Pronunciation Teaching Revisited: A Proposed Measurement Framework and Meta‐Analysis

“…Focusing on naturalistic, advanced‐level L2 learners (rather than beginner–intermediate L2 learners in classroom settings), the existing research on automatized versus implicit knowledge has provided much information, especially to theory building, on the underlying mechanisms of human language acquisition (e.g., clarifying the roles of explicit vs. implicit cognition). In these studies, the type of task modality that can be sensitive enough to capture the theoretically vital distinction between automatized and implicit knowledge has remained controversial (see Plonsky et al., ; Suzuki & DeKeyser, ; Vafaee, Suzuki, & Kachisnke, ).…”

“…In so doing, our goal was to compare the publication trends over the past 10 years (2008–2017), compared to the marginalization observed in and prior to 2005, and to examine the relative status of pronunciation teaching research within a broader framework of instructed SLA research. Ultimately, we aimed to reveal the extent to which L2 pronunciation research has grown and how its growth has aligned with methodological rigor in accessible, peer‐reviewed literature most likely to have an influence on SLA theory and practice (see rationales for similar decisions in Marsden, Morgan‐Short, Thompson, & Abugaber, ; Marsden, Thompson, & Plonsky, ; Norris & Ortega, ; Plonsky, Marsden, Crowther, Gass, & Spinner, ; Spada & Tomita, ).…”

Is Adult Second Language Acquisition Defective?

“…Furthermore, a number of studies (e.g., Johnson, 1992;Bialystok and Miller, 1999;Jia et al, 2002;Clahsen et al, 2010;Spada et al, 2015;Shiu et al, 2018) have found that L2 learners performed better on written than on sGJTs. However, a recent review (Plonsky et al, 2019) suggests that the effect is weak and inconsistent, with some studies reporting an advantage for spoken stimuli.…”

“…Because of these findings, adult second language learning is commonly viewed as "fundamentally different" from child acquisition and defective in some way (see, for example Bley-Vroman, 1989, 2009DeKeyser, 2000;Han and Selinker, 2005;Han, 2013). The failure of many adult L2 learners to acquire native-like linguistic representations is often attributed to maturational changes in the brain, such as lack of or incomplete access to UG (Bley-Vroman, 1989;Schachter, 1996), or less effective procedural learning (DeKeyser, 2000;Paradis, 2004;Granena and Long, 2013;Ullman, 2015). However, alternative explanations, which appeal to the quantity and quality of input available to the learner (Flege et al, 1999;Flege, 2019) and identification with the L2 community (Schumann, 1986;Preston, 1989) have also been put forward.…”