Enhancing topic-detection in computerized assessments of constructed responses with distributional models of language

Martínez-Huertas, José Ángel; Olmos, Ricardo; Gascón, José Antonio León

doi:10.1016/j.eswa.2021.115621

Cited by 4 publications

(11 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1). This method is similar to other previous methods for content-detection tasks such as partial content similarity or partial golden summaries (Dessus & Lemaire, 1999;Franzke et al, 2005;Kintsch et al, 2007;Magliano & Graesser, 2012;Martínez-Huertas et al, 2021). This procedure results in a vector having different numbers of nouns for each concept to be evaluated.…”

Section: Methodsmentioning

confidence: 92%

“…of such text fragments. The essence of this procedure is similar to other previous methods performing content-detection tasks such as partial content similarity or partial golden summaries (explained previously) that select relevant fragments of the original text stimuli (Dessus & Lemaire, 1999;Franzke et al, 2005;Kintsch et al, 2007;Magliano & Graesser, 2012;Martínez-Huertas et al, 2021), but it only uses nouns embedded in those fragments of the instructional text. In this version, a β matrix is created using different k vectors that are compounded by different number of nouns for each concept to be evaluated.…”

Section: Mapping Assessment Rubrics Into Vector Spaces Using the Inbuilt Rubric Methodsmentioning

confidence: 93%

“…This method endows the dimensions of the vector space with semantic meanings determined a priori by the designer of the rubric . This method further makes vectors more than a meaningless set of coordinates as it generates comparable and interpretable coordinates in the vector space (Jorge-Botana et al, 2019 ; Martínez-Huertas et al, 2018 , 2019 , 2021 ; Olmos et al, 2014 , 2016 ).…”

Section: Introductionmentioning

confidence: 99%

“…In psychoeducational assessments, distances between the vectors of student summaries and gold summaries (ideal summaries written by experts) are computed. Golden summaries can be different depending on the method used, as they can be whole gold summaries (ideal summaries to compare with) or partial summaries (paragraphs or sentences extracted from whole gold summaries describing different topics; Dessus & Lemaire, 1999 ; Franzke et al, 2005 ; Kintsch et al, 2007 ; Magliano & Graesser, 2012 ; Martínez-Huertas et al, 2021 ), or a set of gold summaries pre-rated with scores from different expert (Olmos et al, 2009b ). In the field of automated summary evaluation, León et al ( 2006 ) analyzed six different LSA methods.…”

Section: Introductionmentioning

confidence: 99%

“…The scores or coordinates of each dimension show the extent to which a text covers the knowledge domain pertaining to each item. Inbuilt Rubric method can be considered a model that maps assessment rubrics into vector space models (a more specific explanation of this method can be found in Olmos et al, 2014 , 2016 , or in Martínez-Huertas et al, 2021 ). Previous research has shown that the overall scores of the Inbuilt Rubric method demonstrate better performance than the Golden Summary method using the same original LSA semantic space (Martínez-Huertas et al, 2018 , 2019 ; Olmos et al, 2016 ).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables

et al. 2022

Self Cite

View full text Add to dashboard Cite

In this paper, we highlight the importance of distilling the computational assessments of constructed responses to validate the indicators/proxies of constructs/trins using an empirical illustration in automated summary evaluation. We present the validation of the Inbuilt Rubric (IR) method that maps rubrics into vector spaces for concepts’ assessment. Specifically, we improved and validated its scores’ performance using latent variables, a common approach in psychometrics. We also validated a new hierarchical vector space, namely a bifactor IR. 205 Spanish undergraduate students produced 615 summaries of three different texts that were evaluated by human raters and different versions of the IR method using latent semantic analysis (LSA). The computational scores were validated using multiple linear regressions and different latent variable models like CFAs or SEMs. Convergent and discriminant validity was found for the IR scores using human rater scores as validity criteria. While this study was conducted in the Spanish language, the proposed scheme is language-independent and applicable to any language. We highlight four main conclusions: (1) Accurate performance can be observed in topic-detection tasks without hundreds/thousands of pre-scored samples required in supervised models. (2) Convergent/discriminant validity can be improved using measurement models for computational scores as they adjust for measurement errors. (3) Nouns embedded in fragments of instructional text can be an affordable alternative to use the IR method. (4) Hierarchical models, like the bifactor IR, can increase the validity of computational assessments evaluating general and specific knowledge in vector space models. R code is provided to apply the classic and bifactor IR method.

show abstract