Crowdsourced Adaptive Comparative Judgment: A Community‐Based Solution for Proficiency Rating

Abstract

The main objective of this Methods Showcase Article is to show how the technique of adaptive comparative judgment, coupled with a crowdsourcing approach, can offer practical solutions to reliability issues as well as to address the time and cost difficulties associated with a text-based approach to proficiency assessment in L2 research. We showcased this method by reporting on the methodological framework implemented in the Crowdsourcing Language Assessment Project and by presenting the results of a first study that demonstrated that a crowd is able to assess learner texts with high reliability. We found no effect of language skills or language assessment experience on the assessment task, but judges who had received formal language assessment training seemed to differ in their decisions from judges who had not received such training. However, the scores generated by the crowdsourced task exhibited a strong positive correlation with the rubric-based scores provided with the learner corpus used. Keywords learner corpus; language assessment; proficiency; adaptive comparative judgment; crowdsourcingThe Crowdsourcing Language Assessment Project (CLAP) project was developed within the framework of the Lexicogrammatical Complexity Across Mode T.0086.18 FNRS project. We thank Alex König (then at EURAC, Bolzano, Italy) for his technical help at the start of the project. We are grateful to our colleagues at UCLouvain for taking part in the pilot study and to all national and international colleagues who contributed to CLAP and/or provided feedback on the project. We also thank the reviewers for their very constructive and insightful comments. The usual disclaimers apply. We have no known conflict of interest to disclose.

(Why) Are Open Research Practices the Future for the Study of Language Learning?

“…For example, signatories to the Peer Reviewers’ Openness Initiative (Morey et al., 2016) refuse to review papers that do not have open materials and data. Other community‐driven practices draw on crowdsourcing, citizen science, and other types of collaborative and multisite working that can occur in a surprising number of stages of the entire research process, including research team building (e.g., Moshontz et al., 2018), design decisions (e.g., Landy et al., 2020), data collection (e.g., Paquot et al., 2022, validation of community judgements of proficiency), community augmented meta‐analyses that accumulate multiple datasets (see Many Babies Metalab, https://langcog.github.io/metalab/documentation/using_ma_data/contribute_ma), and analysis (Aczel et al., 2021, to improve analytical robustness). Even the writing process itself has attracted an open community‐driven approach (e.g., Tennant et al.’s, 2020, discussion of massively open online papers [MOOPs] that involve between 10 and 100 [partially] self‐selecting authors in an openly participatory format).…”

Crowdsourcing ratings for single lexical items

“…The other is a relatively new topic connected to the reliability of crowdsourcing non-expert judgments as a method of producing linguistic annotations (e.g. Paquot et al, 2022;Alfter et al, 2021).…”

“…This again lines up with previous research investigating the reliability of non-experts in tasks normally requiring expert knowledge. Paquot et al (2022) set essay assessment into a comparative judgment paradigm, employing both trained assessors (experts) and non-trained academics (non-experts). The results clearly show that the two groups exhibit high similarity in their assessments, thus demonstrating that an untrained crowd can be used reliably for essay assessment tasks.…”

“…The short overview presented above demonstrates the use of non-experts for annotation tasks on different linguistic levels: multiword expressions and collocations (Alfter et al, 2021;Kosem et al, 2018), sentences (Lau et al, 2014;Alfter et al, 2022) and texts (De Clerq et al, 2014;Paquot et al, 2022). A number of these studies have contrasted the use of non-experts and experts, demonstrating that the task design has a crucial impact on the reliability of the annotation results, and in particular that the setting of comparative judgments (e.g.…”

Review of Gudmestad, Aarnes; Edmonds, Amanda (Eds.) (2018). Critical Reflections on Data in Second Language Acquisition. John Benjamins

“…Learner texts are therefore not labelled as 'intermediate' or 'advanced', but are rather situated on an overall ranking, thus reflecting the continuous nature of proficiency assessments. In this vein, I wish to also refer the reader to the Crowdsourcing Language Assessment Project (CLAP) (Paquot, Rubin & Vandeweerd, 2022) carried out at UCLouvain (Belgium), which specifically relies on crowdsourcing tools similar to Comproved to study how they can contribute to improving L2 proficiency assessment practices. I strongly urge all researchers who are directly or indirectly concerned with proficiency-related methodological issues to take Leal's arguments on board and start departing from dichotomization practices.…”