Outliers in L2 Research in Applied Linguistics: A Synthesis and Data Re-Analysis

Abstract

Data from self-paced reading (SPR) tasks are routinely checked for statistical outliers (Marsden, Thompson, & Plonsky, 2018). Such data points can be handled in a variety of ways (e.g., trimming, data transformation), each of which may influence study results in a different manner. This two-phase study sought, first, to systematically review outlier handling techniques found in studies that involve SPR and, second, to re-analyze raw data from SPR tasks to understand the impact of those techniques. Toward these ends, in Phase I, a sample of 104 studies that employed SPR tasks was collected and coded for different outlier treatments. As found in Marsden et al. (2018), wide variability was observed across the sample in terms of selection of time and standard deviation (SD)-based boundaries for determining what constitutes a legitimate reading time (RT). In Phase II, the raw data from the SPR studies in Phase I were requested from the authors. Nineteen usable datasets were obtained and re-analyzed using data transformations, SD boundaries, trimming, and winsorizing, in order to test their relative effectiveness for normalizing SPR reaction time data. The results suggested that, in the vast majority of cases, logarithmic transformation circumvented the need for SD boundaries, which blindly eliminate or alter potentially legitimate data. The results also indicated that choice of SD boundary had little influence on the data and revealed no meaningful difference between trimming and winsorizing, implying that blindly removing data from SPR analyses might be unnecessary. Suggestions are provided for future research involving SPR data and the handling of outliers in second language (L2) research more generally.

Research in progress: Applied linguistics at Northern Arizona University, USA

“…However, two other types of efforts are also part of this strand. Students and faculty in the program have produced a number of methodological syntheses , which apply meta-analytic techniques as a means to describe and evaluate research and reporting practices related to a given domain or technique such as self-paced reading tasks (Marsden et al, 2018), grammaticality judgment tasks (Plonsky et al, forthcoming), and outlier handling techniques (Nicklin & Plonsky, forthcoming). In addition to teaching semester-long courses as well as shorter workshops and bootcamps, Plonsky has also sought to improve and advance the field's use of meta-analytic procedures.…”

Applying meta-analysis to research on bilingualism: An introduction

“…*K denotes the number of primary studies in the sample; **d and g both represent standardized mean differences; r = correlations; OR = odds ratio a strategy employed by a number of meta-analyses in the field of bilingualism & Schwartz, 2017; Lehtonen et al, 2018;Melby-Lervåg & Lervåg, 2014;Mukadam et al, 2017;Peng et al, 2018). We encourage authors who do so to provide the response rate for the sake of greater transparency (see, e.g., Nicklin & Plonsky, 2020). Whether or not the missing data are provided, in such a situation, meta-analysts might find themselves feeling constrained by the shortcomings or even confounds in the designs and reporting practices of primary research.…”

Open Science, Data Analysis, and Data Sharing

“…This creates a paper trail from the raw data to the data you used in analyses ultimately reported in the article. It also can help explain or avoid situations in which other researchers, working off of the same raw data, end up with different results due to not knowing the data cleaning procedures of the original study (e.g., Nicklin & Plonsky, 2020). Finally, while not quite as necessary as primary analysis and data 13 cleaning scripts, it is helpful to share scripts used to create data-based plots that are part of your article.…”

“…Sharing data can also facilitate integrative data analysis (Hussong et al, 2013), where researchers combine data from several studies (e.g., Isbell & Son, 2021), which in turn can help overcome the issue of small sample sizes in many applied linguistics subdomains (e.g., instructed SLA, Loewen & Hui, 2021). In research syntheses, open data and analyses can help other researchers recover unreported statistics, more accurately code methodological features of studies, and conduct useful reanalyses of data (Nicklin & Plonsky, 2020). Students and fellow researchers alike can also learn from your analyses and practice their skills using your data.…”