Doing L2 Speech Research Online: Why and How to Collect Online Ratings Data

Abstract

Listener-based ratings have become a prominent means of defining second language (L2) users’ global speaking ability. In most cases, local listeners are recruited to evaluate speech samples in person. However, in many teaching and research contexts, recruiting local listeners may not be possible or advisable. The goal of this study was to hone a reliable method of recruiting listeners to evaluate L2 speech samples online through Amazon Mechanical Turk (AMT) using a blocked rating design. Three groups of listeners were recruited: local laboratory raters and two AMT groups, one inclusive of the dialects to which L2 speakers had been exposed and another inclusive of a variety of dialects. Reliability was assessed using intraclass correlation coefficients, Rasch models, and mixed-effects models. Results indicate that online ratings can be highly reliable as long as appropriate quality control measures are adopted. The method and results can guide future work with online samples.

Automated assessment of second language comprehensibility: Review, training, validation, and generalization studies

“…Each session took around 2 hours with a 5-minute intermission a halfway through. In accordance with the recommended quality control measures for online L2 rating data collection (Nagle & Rehman, 2021), the platform was designed so that the listeners had to listen to the full-length of each sample (30 sec) before they rated the comprehensibility. The entire secession was carefully monitored by the investigator using a video-conferencing tool.…”

“…This should be considered as an intriguing direction for future research. data using online platforms to reduce the burden on researchers (e.g., Nagle & Rehman, 2021 for Amazon Mechanical Turk). Therefore, due to the time-consuming nature of the assessment procedure, a growing amount of attention has been given to the idea of using automated L2 comprehensibility scoring (O'Brien et al, 2018).…”

“…However, eliciting listeners’ comprehensibility judgments is a time-consuming task (e.g., about 1 hour for rating 70 samples in Derwing & Munro, 2013). To this end, where some researchers use shorter stimuli to avoid listener fatigue (e.g., 4.5–10.5 seconds in Derwing & Munro, 1997), others recommend collecting L2 rating data using online platforms to reduce the burden on researchers (e.g., Nagle & Rehman, 2021 for Amazon Mechanical Turk). Therefore, due to the time-consuming nature of the assessment procedure, a growing amount of attention has been given to the idea of using automated L2 comprehensibility scoring (O’Brien et al, 2018).…”

Speaking performances, stakeholder perceptions, and test scores: Extrapolating from the Duolingo English test to the university

“…Data quality is a concern in online speech research (Nagle & Rehman, 2021), and the Rasch approach to measurement generally prescribes removing elements with excessively poor fit. As previously mentioned, there were 12 listeners who, based on poor performance on attention checks, may have supplied low-quality (e.g., inattentive) judgment data.…”

Exploring phonetic predictors of intelligibility, comprehensibility, and foreign accent in L2 Spanish speech

“…In a random‐raters design, a unique group of k raters evaluates each sample, such that each sample receives the same number of evaluations, but the rater group for each sample is different. Such designs are common in large‐scale pronunciation evaluations (Peabody, 2011) and have been shown to be viable and reliable in L2 speech research (Nagle & Rehman, 2021). Future research would also benefit from integrating multiple measures of intelligibility, such as performance on nonsense sentences and true–false statements, which appear to be the best predictors of listening comprehension (Kang et al., 2018).…”