Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques

Hsu, Fu Yuan; Lee, Hahn Ming; Chang, Tao; Sung, Yao–Ting

doi:10.1016/j.ipm.2018.06.007

Cited by 28 publications

(23 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, we applied 55 statistically oriented association measures. For future research, we are excited to see that new computational algorithms for learning features embedded in language are burgeoning, such as HAL (Lund & Burgess, 1996), LSA (Landauer & Dumais, 1997), BEAGLE (Jones & Mewhort, 2007), Contextual_SOM (Zhao, Li, & Kohonen, 2011), and most recently, Word2Vec (Hsu, Lee, Chang, & Sung, 2018; Mikolov, Chen, Corrado, & Dean, 2013). We look forward to more studies in the future on the design of new methods of measuring the strength of word associations using corpus data, or on how to transform techniques in related fields, such as NLP or AI, into word association measures.…”

Section: Discussionmentioning

confidence: 99%

CLAD: A corpus-derived Chinese Lexical Association Database

et al. 2019

Self Cite

View full text Add to dashboard Cite

The application of word associations has become increasingly widespread. However, the association norms produced by traditional free association tests tend not to exceed 10,000 stimulus words, making the number of associated words too small to be representative of the overall language. In this study we used text corpora totaling over 400 million Chinese words, along with a multitude of association measures, to automatically construct a Chinese Lexical Association Database (CLAD) comprising the lexical association of over 80,000 words. Comparison of the CLAD with a database of traditional Chinese word association norms shows that word associations extracted from large text corpora are similar in strength to those elicited from free association tests but contain a much greater number of associative word pairs. Additionally, the relatively small numbers of participants involved in the creation of traditional norms result in relatively coarse scales of association measurement, whereas the differentiation of association strengths is greatly enhanced in the CLAD. The CLAD provides researchers with a great supplement to traditional word association norms. A query website at www.chinesereadability.net/LexicalAssociation/CLAD/ affords access to the database.

show abstract

Section: Discussionmentioning

confidence: 99%

CLAD: A corpus-derived Chinese Lexical Association Database

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Perhaps the most famous example is that the word “queen” can be derived from the vector-space word representations of “king,” “man,” and “woman”: The result of the vector subtraction of vec(“king”; i.e., the word vector of “king”) from vec(“man”) and the vector addition of vec(“woman”) is closer to vec(“queen”) than to any other word vector. More recently, researchers have used Word2Vec-based algorithms to identify linguistic relations beyond the word level (i.e., relations at the sentence, paragraph, and discourse levels), with initial successes being reported (Hsu et al, 2018; Tseng et al, 2019; Wang et al, 2016). Some studies have obtained effective results by grouping synonymous phrases and classifying short texts (e.g., Bansal & Srivastava, 2018; Zhang et al, 2015).…”

Section: Approaches To Assessing Creativity and Dt Testsmentioning

confidence: 99%

Construction and validation of a computerized creativity assessment tool with automated scoring based on deep-learning techniques.

Sung¹,

Cheng²,

Tseng³

et al. 2024

Psychology of Aesthetics, Creativity, and the Arts

Self Cite

View full text Add to dashboard Cite

Based on the divergent thinking (DT) framework of creativity assessment, this study constructed the Computerized Creativity Assessment with Figure Test (C-CRAFT) that is equipped with an automated scoring system and built around a deep-learning-based semantic space model called Word2Vec. A subject pool of 493 undergraduates completed the C-CRAFT as well as a conventional paper-and-pencil DT test that required manual scoring. We found moderately high to high coefficients for the correlations between the two tests, which suggested that the C-CRAFT has strong criterion-related validity. The results of the pre-and posttests also demonstrated the high test-retest reliability of the C-CRAFT. Good discriminant validity was evidenced by highly significant differences in the C-CRAFT scores between college students from art and design-related fields and students from other majors. These research findings indicate that the C-CRAFT is a valid and reliable assessment tool for DT, while the automated nature of the C-CRAFT makes it easier to implement the DT test compared with traditional approaches. Moreover, by applying the C-CRAFT to the Chinese language, this study contributes to the cross-linguistic research of semantic models in creativity assessment.

show abstract

“…Machine learning (ML) is an emerging computerized technology that relies on algorithms constructed by "learning" from training data, with the potential to revolutionize science assessments. Hsu et al [13] developed an ML method for automatically estimating the item difficulty of social studies tests that constructed a semantic space based on item elements of multiple-choice items, i.e., test stem, answer, and alternative options. With this, they showed that the new method outperformed the traditionally widespread yet resource-consuming pretesting method.…”

Section: Application Of ML To Length and Difficulty Level Of Items In Science Assessmentsmentioning

confidence: 99%

“…With this, they showed that the new method outperformed the traditionally widespread yet resource-consuming pretesting method. A review study [13] included 49 machine-based studies on science assessments, finding that many studies (24 of 49) directly embedded ML in pedagogical activities. However, no study about the difficulty level concerning the lengths of an item's elements was reported, indicating a research gap in using ML techniques in science assessments.…”

Section: Application Of ML To Length and Difficulty Level Of Items In Science Assessmentsmentioning

confidence: 99%

Association between Test Item's Length, Difficulty, and Students' Perceptions: Machine Learning in Schools' Term Examinations

Lin¹,

Lu²,

Lien³

2021

ujer

View full text Add to dashboard Cite

The study applies machine learning (ML) algorithms to investigate the association between the length of a test item written in Chinese (through word count), item difficulty, and students' item perceptions (IPs) in science term examinations. For Research Question 1, items for grade 7 students aged 12-13 in a Taiwanese secondary school from 2014 to 2019 were analyzed. For Research Question 2, the study included 4,916 students from the said population. For RQ3, perceptions were gathered from 48 students of the same school in 2020. The study's results showed that first, the average word count of the 611 items was 88.81, with an average stem word count of 41.16, average options word count of 47.66, and stem-to-options word count ratio (S-O ratio) of 1.27. Second, given that the ML M5P categorization algorithm affirms the items' predictive power, the length of an item is a key factor in determining its difficulty. As a result of this algorithm, 3 categories of the length of science term examination items were classified (<57.5 words, 57.5-91.5 words, and >91.5 words), and 3 linear prediction models of item difficulty (LM1, LM2, and LM3) were generated. From these models, it was found that as the length of an item increases, so does its difficulty. In the prediction analysis of students' IPs, the J48 prediction result was better and convertible into understandable rules. IP was the root node of the decision rule, indicating the importance of this variable. Therefore, students would more likely answer an item correctly when 1) it was perceived to be easy or normal, 2) the students had high or ordinary learning achievements in science, and 3) it contained below 71 words. The study's results can be used as a reference for educators, examiners, and researchers in practical science term examination design. Moreover, it can guide the further research method and direction of applying machine learning in analyzing the difficulty of items in scientific assessments.

show abstract

Automated estimation of item difficulty for multiple-choice tests: An application of word embedding techniques

Cited by 28 publications

References 27 publications

CLAD: A corpus-derived Chinese Lexical Association Database

CLAD: A corpus-derived Chinese Lexical Association Database

Construction and validation of a computerized creativity assessment tool with automated scoring based on deep-learning techniques.

Association between Test Item's Length, Difficulty, and Students' Perceptions: Machine Learning in Schools' Term Examinations

Contact Info

Product

Resources

About