Candidate evaluation strategies for improved difficulty prediction of language tests

Abstract

Language proficiency tests are a useful tool for evaluating learner progress, if the test difficulty fits the level of the learner. In this work, we describe a generalized framework for test difficulty prediction that is applicable to several languages and test types. In addition, we develop two ranking strategies for candidate evaluation inspired by automatic solving methods based on language model probability and semantic relatedness. These ranking strategies lead to significant improvements for the difficulty prediction of cloze tests.

Entropy as a Proxy for Gap Complexity in Open Cloze Tests

“…Beinborn et al (2014) built models to predict the difficulty of C-tests (i.e. gaps with half of the required word removed) at the gap and test level and later extended their approach to cover closed cloze tests (Beinborn et al, 2015;Beinborn, 2016). More recently, Pandarova et al (2019) presented a difficulty prediction model for cued gap-fill exercises aimed at practising English verb tenses while Lee et al (2019) investigated how difficulty predictions could be manipulated to adapt tests to a target proficiency level.…”

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

“…The effect of noisy labels on machine learning algorithms has been extensively studied in terms of their effect on system training in both general machine learning literature (see, for example, Frénay and Verleysen (2014) for a comprehensive review), NLP (Reidsma and Carletta, 2008;Beigman Klebanov and Beigman, 2009;Schwartz et al, 2011;Plank et al, 2014;Martínez Alonso et al, 2015;Jamison and Gurevych, 2015) and automated scoring (Horbach et al, 2014;Zesch et al, 2015).…”

“…Ever since the release of Kaggle's Automatic Student Assessment Prize's (ASAP) Automatic Essay Grading (AEG) dataset in 2012, there has been a lot of work on holistic essay grading. Initial approaches, such as those of Phandi et al (2015) and Zesch et al (2015) made use of machine learning techniques in scoring the essays. A number of other works used various deep learning approaches, such as Long Short Term Memory (LSTM) Networks Tay et al, 2018) and Convolutional Neural Networks (CNN) (Dong and Zhang, 2016;).…”

“…In our paper, we compare three approaches to score essay traits, which are trait agnostic. The first uses a set of task-independent features as described by Zesch et al (2015) and Mathias and Bhattacharyya (2018a). The second uses a string kernel-base approach as well as super word embeddings as described by .…”

Predicting the Difficulty of Multiple Choice Questions in a High-stakes Medical Exam

“…There have also been attempts to estimate the difficulty of questions for humans. This has been mostly done within the realm of language learning, where the difficulty of reading comprehension questions is strongly related to their associated text passages (Huang et al, 2017;Beinborn et al, 2015;Loukina et al, 2016). Another area where question-difficulty prediction is discussed is the area of automatic question generation, as a form of evaluation of the output (Alsubait et al, 2013;Ha and Yaneva, 2018).…”