Entropy as a Proxy for Gap Complexity in Open Cloze Tests

Felice, Mariano; Buttery, Paula

doi:10.26615/978-954-452-056-4_037

Cited by 10 publications

(9 citation statements)

References 12 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For each gap, we generate a sentence where only the gap is replaced by the masking token and fetch its predictions from the BERT model. From these predictions we take the prediction probability of the solution as the first feature and the entropy of the prediction probabilities of the top-50 predicted words as the second feature in concordance with findings by Felice and Buttery (2019) who show that entropy strongly correlates with the gap difficulty. Adding both features to the 59 features proposed by Beinborn (2016) increases the accuracy of our MLP from 0.33 to 0.37.…”

Section: Datasupporting

confidence: 71%

Empowering Active Learning to Jointly Optimize System and User Demands

Lee¹,

Meyer²,

Gurevych³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Existing approaches to active learning maximize the system performance by sampling unlabeled instances for annotation that yield the most efficient training. However, when active learning is integrated with an end-user application, this can lead to frustration for participating users, as they spend time labeling instances that they would not otherwise be interested in reading. In this paper, we propose a new active learning approach that jointly optimizes the seemingly counteracting objectives of the active learning system (training efficiently) and the user (receiving useful instances). We study our approach in an educational application, which particularly benefits from this technique as the system needs to rapidly learn to predict the appropriateness of an exercise to a particular user, while the users should receive only exercises that match their skills. We evaluate multiple learning strategies and user types with data from real users and find that our joint approach better satisfies both objectives when alternative methods lead to many unsuitable exercises for end users. 1

show abstract

Section: Datasupporting

confidence: 71%

Empowering Active Learning to Jointly Optimize System and User Demands

Lee¹,

Meyer²,

Gurevych³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…A method of evaluating open cloze tests using entropy to model the restrictiveness of the context provided around the target word was proposed [11]. The context surrounding target words in open cloze tests requires a limited context to reduce choices so students do not respond with unexpected answers that are technically correct.…”

Section: Related Workmentioning

confidence: 99%

“…This study uses the number of possible syntactically and semantically correct choices for a gap and the probability of those choices to measure the restrictiveness of the context provided by questions. The number of choices and their probability is modeled with entropy, "which quantifies the amount of information conveyed by an event" [11]. 5-gram bi-directional language model was used to measure entropy.…”

Section: Related Workmentioning

confidence: 99%

“…Previous studies [10], did not use actual language modeling but did use a method similar to n-gram statistical language modeling, as counting the number of times an n-gram appears in Google N-grams does approximate the probability of that n-gram. [11] use language modeling to measure both candidate space and candidate probability for use in calculating entropy, which models the quality of the context provided by the text. Study by [13] uses language modeling directly to predict difficulty of a text: when a language could not predict a word, that correlated with a higher percentage of students being unable to predict the word.…”

Section: Proposed Approach: Language Modeling As Student Modelingmentioning

confidence: 99%

See 1 more Smart Citation

Generating Vocabulary Sets for Implicit Language Learning using Masked Language Modeling

Edgar¹

2022

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

A well-balanced language curriculum must include both explicit vocabulary learning and implicit vocabulary learning. However, most language learning applications focus on explicit instruction. Students require support with implicit vocabulary learning because they need enough context to guess and acquire new words. Traditional techniques aim to teach students enough vocabulary to comprehend the text, thus enabling them to acquire new words. Despite the wide variety of support for vocabulary learning offered by learning applications today, few offer guidance on how to select an optimal vocabulary study set. This paper proposes a novel method of student modeling with masked language modeling to detect words that are required for comprehension of a text. It explores the efficacy of using deep learning via a pre-trained masked language model to model human reading comprehension and presents a vocabulary study set generation pipeline (VSGP). Promising results show that masked language modeling can be used to model human comprehension and the pipeline produces reasonably sized vocabulary study sets that can be integrated into language learning systems.

show abstract

“…In these cases, tests are dynamically adapted to the examinee's proficiency level during the test session. From a different perspective, Felice and Buttery (2019) show that controlling gap entropy can be useful for designing open cloze tests at different CEFR levels. The work we present in this paper, however, aims to model the more complex task of predicting a full set of gaps at the paragraph level that comply with design and testing principles and is, to the best of our knowledge, the first to employ and adapt transformer-based models for this task.…”

Section: Related Workmentioning

confidence: 99%

Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers

Felice¹,

Taslimipoor²,

Buttery³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents the first multi-objective transformer model for constructing open cloze tests that exploits generation and discrimination capabilities to improve performance. Our model is further enhanced by tweaking its loss function and applying a post-processing reranking algorithm that improves overall test structure. Experiments using automatic and human evaluation show that our approach can achieve up to 82% accuracy according to experts, outperforming previous work and baselines. We also release a collection of highquality open cloze tests along with sample system output and human annotations that can serve as a future benchmark.1 Now known as B2 First: https://www.cambridge english.org/exams-and-tests/first/ 2 Dataset available at https://github.com/Cambr idgeALTA/fce-cep-oc.

show abstract

Entropy as a Proxy for Gap Complexity in Open Cloze Tests

Cited by 10 publications

References 12 publications

Empowering Active Learning to Jointly Optimize System and User Demands

Empowering Active Learning to Jointly Optimize System and User Demands

Generating Vocabulary Sets for Implicit Language Learning using Masked Language Modeling

Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers

Contact Info

Product

Resources

About