Multisite Replication in Second Language Acquisition Research: Attention to Form During Listening and Reading Comprehension

Abstract

We conducted a multisite replication study with aspects of preregistration in order to explore the feasibility of such an approach in second language (L2) research. To this end, we addressed open questions in a line of research that has examined whether having learners attend to form while reading or listening to a L2 passage interferes with comprehension. Our results are consistent with findings from the specific paradigm that we replicated in that no effects on comprehension were detected in analyses conducted over all sites. However, further investigation is warranted due to site‐specific effects and methodological limitations. We found all aspects of the multisite registered replication approach to be useful although the registration component itself appeared to be an especially feasible and valuable first step toward increasing the robustness and generalizability of findings in our field. Open Practices This article has been awarded Open Data, Open Materials, and Preregistered Research Design badges. The following information is publicly accessible via the Open Science Framework: registered materials and protocol (https://osf.io/tvuer), registered data entry template (https://osf.io/d5s2t), open data (https://osf.io/vwytd), and open analysis (https://osf.io/nz3su). Learn more about the Open Practices badges from the Center for Open Science: https://osf.io/tvyxz/wiki.

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

“…This reticence is complex. Anecdotally, we observed during colloquia discussing this study and the research by Morgan-Short et al (2018) that some researchers reported actively undertaking and promoting replication with students and in their own work, yet they were less enthusiastic about labeling these studies as replications. Here we illustrate with three relatively recent examples of what we think is fairly standard practice.…”

“…Effect sizes, as authors have noted many times (e.g., Norris et al, 2015), are useful because they enable comparisons to be made using standardized units across studies to interpret the magnitude of difference or association in meaningful paired comparisons. Morgan-Short et al (2018) provided an example of a study giving independent effect sizes for intersite comparisons and aggregated effect sizes in an intrastudy meta-analysis of direct replications (see also Ellis & Sagarra, 2011). In our sample of 70 initial studies, Cohen's d was provided in seven studies and r by one study, whereas 81% did not provide any effect size values.…”

“…The large multisite efforts have thus far been in fields where large authorship teams are the norm. In line with these practices, Morgan-Short et al (2018) offered coauthorship to those collecting and entering data and running predefined analyses, with lead authorship for those convening the multisite replication, providing the protocols, and formally reporting the results. Even with this coauthorship agreement, they were fortunate in securing collaborators, and a reciprocal ethic is needed to support such large-scale multisite replication efforts (such as "I collect data for others; others collect data for me").…”

“…We found only one bundle of self-labeled replications that conducted an internal meta-analysis, which was possible because the researchers used the same materials and had the data from the initial study fully available (Ellis et al, 2014; see also Lindsay, 2017;Morgan-Short et al, 2018). Making data available entails ethical considerations (e.g., institutional review boards) early in the research process and is not possible in all situations, but it is increasingly a requirement of funders.…”

“…On the other hand, even studies that fall into a narrower definition of replication (e.g., investigating the same research questions with a design and materials that are as similar as possible to an earlier study) may not label themselves as replications. To illustrate, of the four studies that were part of the replication sequence extended by Morgan-Short et al (2018), only one (Wong, 2001) turned up in our systematic review as a self-labeled replication. Given this subjectivity and inconsistency and, more importantly, given that we wanted in our systematic review to ascertain the extent to which the term replication has been used to label studies reported in journals, we used instead the self-identification of authors, that is, studies that self-labeled in the title or abstract as a replication study.…”