Study Quality in Quantitative L2 Research (1990–2010): A Methodological Synthesis and Call for Reform

Abstract

This article builds on the growing line of inquiry into methodological practices in quantitative second language (L2) research. Specifically, the study uses synthetic techniques to examine changes over time in research and reporting practices. 606 primary reports of quantitative L2 research from two journals—Language Learning and Studies in Second Language Acquisition—were surveyed on different design features, statistical analyses, and data reporting practices. Frequencies and percentages of each feature were then calculated and compared across the 1990s and the first decade of the 2000s to examine changes taking place in the field. The results, while not necessarily representative of all substantive and methodological approaches within the domain of L2 research, indicate numerous changes including increases in sample sizes, delayed posttesting, and the availability of critical data such as effect sizes, reliability estimates, and standard deviations to accompany means. With respect to statistical procedures, the range of analyses has not changed, and the field continues its unfortunate reliance on statistical significance. The findings are grouped according to three themes, which are discussed in light of previous reviews in this and other fields: (a) means‐based analyses, (b) missing data, null hypothesis significance testing, and the “power problem,” and (c) design preferences. The article concludes with an extended call for reform targeting six groups of stakeholders in the field. Most notably, an argument is made for field‐specific methodological standards and enhancements to graduate curricula and training.

A meta-analysis: Age and second language grammar acquisition

“…Reporting a p-value as less than 0.5 might be problematic for two reasons. One, because of a smaller sample size for each group (23), there is a higher susceptibility to sampling error and hence, less reliable effect size; and two, small effect sizes for small samples might be overestimated (Plonsky, 2014).…”

Methodological synthesis of research on the effectiveness of corrective feedback in L2 writing

“…Coming out of the metaanalytic tradition (Norris & Ortega, 2000, Norris & Ortega, 2006, methodological review allows for systematic description of research and reporting practices within a domain for the purpose of guiding future research efforts. Recent studies have synthesized quantitative L2 research for purposes of reflection and advancement in research practices across the field in general (Plonsky, 2013;Plonsky, 2014a) and within subdomains (e.g., interaction; Plonsky & Gass, 2011). Studies in the area have largely addressed reporting and sampling practices, major design types, and statistical analyses.…”

Research Methodology in Second Language Studies: Trends, Concerns, and New Directions

“…Looking at generalizability from another, layered perspective, we have seen a relatively recent emergence of emphasis on synthesis and meta‐analysis in the more cognitively oriented approaches. This is allowing applied linguistics to address broader research questions than those in the original, individual reports (Norris & Ortega, ; Plonsky, , ; Plonsky & Oswald, ). Meta‐analyses are helping to define the next generation of research questions as well as provide helpful answers to the ones already asked.…”