DeepBlueAI at SemEval-2021 Task 1: Lexical Complexity Prediction with A Deep Ensemble Approach

Abstract

Lexical complexity plays an important role in reading comprehension. lexical complexity prediction (LCP) can not only be used as a part of Lexical Simplification systems, but also as a stand-alone application to help people better reading. This paper presents the winning system we submitted to the LCP Shared Task of SemEval 2021 that capable of dealing with both two subtasks. We first perform fine-tuning on numbers of pre-trained language models (PLMs) with various hyperparameters and different training strategies such as pseudo-labelling and data augmentation. Then an effective stacking mechanism is applied on top of the fine-tuned PLMs to obtain the final prediction. Experimental results on the Complex dataset show the validity of our method and we rank first and second for subtask 2 and 1. Multi-wordsContext1: SEM confirmed many of the observations made by confocal microscopy. Complexity score: 0.64473 Context2: SJ and SVJ carried out confocal microscopy on whole-mounts of stria vascularis. Complexity score: 0.7750 Single word Context1:They shall be to you for a refuge from the avenger of blood. Complexity score: 0.3475 Context2: There will be a pavilion for a shade in the daytime from the heat, and for a refuge and for a shelter from storm and from rain.

An Evaluation of Binary Comparative Lexical Complexity Models

“…We trained a SVM model given its high performance at binary CWI Choubey and Pateria, 2016;Sanjay et al, 2016;Kuru, 2016), a BERT model (Devlin et al, 2019) per its competitive performance at LCP-2021 (Shardlow et al, 2021;Yaseen et al, 2021;Pan et al, 2021;Rao et al, 2021), and a BERT + multi-layer perceptron (MLP) model (Gu and Budhkar, 2021) to take full advantage of BERT inferred contextual features as well as the wordlevel features fed into our SVM model. Two naive baseline models were used to evaluate the performances of our SVM, BERT, and BERT + MLP models: a random classifier (RC) and a majority classifier (MC).…”

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

“…We trained a SVM model given its high performance at binary CWI Choubey and Pateria, 2016;Sanjay et al, 2016;Kuru, 2016), a BERT model per its competitive performance at LCP-2021 Yaseen et al, 2021;Pan et al, 2021;Rao et al, 2021), and a BERT + multi-layer perceptron (MLP) model (Gu and Budhkar, 2021) to take full advantage of BERT inferred contextual features as well as the wordlevel features fed into our SVM model. Two naive baseline models were used to evaluate the performances of our SVM, BERT, and BERT + MLP models: a random classifier (RC) and a majority classifier (MC).…”

“…Gooding and Kochmar, 2018;Kajiwara and Komachi, 2018). The 2021 shared task was won by Pan et al (2021) using an ensemble of pre-trained Transformers , but submissions with feature-based models also ranked highly (Mosquera, 2021;Rotaru, 2021).…”

“…However, due to the limitation of the current QA corpus, most of the generated questions focus on finding the information presented in the passage. Moreover, the majority of NQG models are used for improving QA or dialogue systems instead of for educational purposes (Duan et al, 2017;Sachan and Xing, 2018;Pan et al, 2021).…”

Designing and Evaluating a User Interface for People with Cognitive Disabilities

“…Table 2.7 shows a brief description of the task's participants. One of the participants with the best scores was the DeepBlueAI [137] team by using a wide variety of pre-trained language models along with different training strategies such as pseudo-labelling and data augmentation to finally apply a stacking method to give the final prediction. With these methods, the team obtained the highest "Pearson's Correlation" in the second task, and the second-best in the first task.…”