Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Abstract

Despite its critical role for the development of the field, little is known about replication in second language (L2) research. To better understand replication practice, we first provide a narrative review of challenges related to replication, drawing on recent developments in psychology. This discussion frames and motivates a systematic review, building on syntheses of replication in psychology, education, and L2 research. We coded 67 self-labeled L2 replication studies found across 26 journals for 136 characteristics. We estimated a mean rate of 1 published replication study for every 400 articles, with a mean of 6.64 years between initial and replication studies and a mean of 117 citations of the initial study before a replication was published. Replication studies had an annual mean of 7.3 citations, much higher than averages in linguistics and education. Overlap in authorship between initial and replication studies and the availability of the initial materials both increased the likelihood of a replication supporting the initial findings. Our sample contained no direct (exact) replication attempts, and changes made to initial studies were numerous and wide ranging, which likely obscured, if not undermined, the interpretability of replication studies. To improve the amount and quality of L2 replication research, we propose 16 recommendations relating to rationale, nomenclature, design, infrastructure, and incentivization for collaboration and publication.

Do Explicit Instruction and High Variability Phonetic Training Improve Nonnative Speakers’ Mandarin Tone Productions?

“…() may have partially reflected the additional input participants received outside of the experiment during their L2 classroom learning. The difference between the two studies highlights the value in using a variety of stimuli and tasks in assessing native and nonnative speech (e.g., Marsden et al., ; Perrachione et al., ).…”

Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences

“…We briefly discuss two approaches to addressing this combined problem of small data sets and lack of replication, both of which set the scene for the introduction of Registered Reports at Language Learning . First, although technological developments (such as platforms to support preregistration, open materials, data, and software) facilitate multisite replication projects that gather large data sets (e.g., Morgan‐Short, Marsden, Heil, et al., ), the perceived extra effort these approaches require can deter researchers, especially given the lack of assurance of eventual publication. Second, small samples in human participant research may be unavoidable due to the limited resources available to many researchers.…”

“…Small data sets are less problematic under a more synthetic research ethic, where replications are synthesized within primary research that combines new data with previous data sets via meta‐analysis (e.g., Ellis & Sagarra, ; Morgan‐Short et al., ). In actuality, however, meta‐analyses are frequently secondary in that they synthesize previously published studies (Plonsky & Brown, ).…”

“…To illustrate, in the domain of second language research, Marsden et al. () reported fewer than one self‐labeled replication study in every 400 journal articles. They also noted an absence of direct replications and a great deal of heterogeneity (often unacknowledged or unjustified by the research aims) between self‐labeled replications and the studies they replicated, which undermines comparisons between studies.…”

Open science in second language acquisition research: The IRIS repository of research materials and data

“…Replication research allows us to investigate the external validity (generalizability) of findings in, for example, different settings, demographics, target languages, helping researchers to test theories, constructs, and measurements (Marsden, Morgan-Short, Thompson, & Abugaber, 2018;Porte, 2012). It can also inform us about the reliability of research and of measures with the same or different samples of learners (Plonsky & Derrick, 2016).…”