Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)

Abstract

Applied linguists have turned increasingly in recent years to meta-analysis as the preferred means of synthesizing quantitative research. The first step in the meta-analytic process involves defining a domain of interest. Despite its apparent simplicity, this step involves a great deal of subjectivity on the part of the meta-analyst. This article problematizes the importance of clearly defining and operationalizing meta-analytic domains. Toward that end, we present a critical review of one particular domain, corrective feedback, which has been subject to 18 unique meta-analyses. Specifically, we examine the unique approach each study has taken in defining their domain of interest. In order to demonstrate the critical role of this stage in the meta-analytic process, we also examine variability in summary effects as a function of the unique subdomains in the sample. Because different techniques used to identify candidate studies carry assumptions about the type of research that falls within the domain of interest (e.g. published vs. unpublished), we also include a brief review of search techniques employed in a set of 81 meta-analyses of second language research. Building on the work of In’nami and Koizumi (2010) and Oswald and Plonsky (2010), the results for this phase of the analysis show that L2 meta-analysts generally rely on a stable but very limited set of search strategies, none of which is likely to yield unpublished studies. Based on our findings related both to domain definition and search techniques employed by L2 researchers, we make specific recommendations for future meta-analytic practice in the field.

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

“…However, it remains worrying that our sample only provided two clusters of studies that self-labeled as overt sequences of an agenda that extended beyond two studies (the VanPatten-Cadierno-DeKeyser cluster and the Ellis-Sagarra et al cluster). Among other concerns, it suggests that the many syntheses and meta-analyses in the field (e.g., Plonsky & Brown, 2015, examined 81 meta-analyses) are bringing together studies that did not self-identify as replications of any kind. Meta-analysts seem to have observed this issue frequently because they have commented on the less-than-ideal comparability between studies in the domain under investigation (due to inconsistency of materials, measures, etc.)…”

Methodological synthesis of research on the effectiveness of corrective feedback in L2 writing

“…Ferris (2004) remarked how little is reported clearly about students' proficiency, their writing background, or formal grammar knowledge. Critics have also noted the lack of description concerning affective factors or details of classroom procedures that could influence attitudes in receiving and dealing with feedback and motivation for improving accuracy, such as grading of assignments or incentives for participating in research (Bruton, 2009(Bruton, , 2010Gué nette, 2007;Plonsky & Brown, 2015). Additionally, details of the treatment are often lacking, such as vague reporting of whether additional feedback on content or clarity is included along with the treatment (Truscott, 2010), or reporting of the length of writings, a variable that Storch (2010) suggests may moderate effectiveness of different CF types.…”

“…We searched the most common academic databases used in meta-analyses in applied linguistics (Plonsky & Brown, 2015) -Education Resources Information Center (ERIC), the Linguistic and Language Behavior Abstracts (LLBA), and ProQuest's Dissertations and Theses using the following combination of keywords corrective feedback OR error correction OR error feedback, AND second language (OR L2), AND writing (OR written), AND accuracy (OR grammar). We also conducted an ancestry search by examining the reference lists of previous review papers (e.g., Ferris, 2004;Storch, 2010;Van Beuningen, 2010) as well as the meta-analyses on WCF (Biber et al, 2011;Russell & Spada, 2006;Truscott, 2007) in order to identify any articles that may have been unavailable through the database search.…”

The type and linguistic foci of oral corrective feedback in the L2 classroom: A meta-analysis

“…Classroom observational research also investigates student affect in response to CF, noticing or awareness of CF, or how teacher beliefs influence the provision of CF, among other foci. In recent years a growing number of meta-analyses (18 to date; Plonsky & Brown, 2014) have synthesized various domains of CF research (e.g. oral, written, computer mediated) with general findings that lend substantial support to the efficacy of CF in L2 learning.…”