Transforming LSA space dimensions into a rubric for an automatic assessment and feedback system

Analyzing Two Automatic Latent Semantic Analysis (LSA) Assessment Methods (Inbuilt Rubric vs. Golden Summary) in Summaries Extracted from Expository Texts

“…A single grade is obtained, as a function of the semantic distance between the student's text and the expert criterion. Golden Summary has frequently been used to evaluate summaries but a major limitation of this method is that the vector representation of a student summary does not capture the main ideas included in its answer; rather all the ideas are collected in a single vector (Olmos, Jorge-Botana, León, & Escudero, 2014;Olmos, Jorge-Botana, Luzón, Cordero, & León, 2016). To solve this limitation, Franzke, Kinstch, Caccamise, Johnson, and Dooley (2005) proposed an elaboration of partial golden summaries designed to detect students' misconceptions via thresholds, but the problem remains because the student's vector is conceived as a whole with the consequence that it is not easy to detect different ideas in the student's vectors.…”

“…The Inbuilt Rubric method is a new method that accommodates a conceptual rubric in the LSA in order to detect contents more precisely and to overcome the limitations of the Golden Summary methods (Olmos et al, 2016). This method identifies the main contents of a text.…”

“…The k main concepts are represented in the LSA semantic space as k vectors. The last step consists of transforming the original latent semantic space to a semantic space where the first k dimensions have the meaning of the main concepts of the instructional text (a complete explanation of the method can be seen in Olmos et al, 2014;Olmos et al, 2016). Thus, the idea of the Inbuilt Rubric method is that the original semantic latent space, where the dimensions are meaningless is transformed into a new semantic space whose first k dimensions now capture the conceptual axes of the rubric.…”

“…Other studies have found that some students write summaries with many irrelevant words (e.g., Olmos et al, 2016). For this reason, we introduced a second condition in the Inbuilt Rubric method: we compared a weighted and a non-weighted version of this method.…”

“…In short, the aim of this study was twofold: a) to compare two automatic assessment methods using LSA for a student's assessment summaries -a classical method, Golden Summary (e.g., Foltz et al, 1999;Klein et al, 2011;Landauer et al, 1998) in which each student's summary was compared to an ideal summary created by experts, and Inbuilt Rubric (Olmos et al, 2014;Olmos et al, 2016), that accommodates an expert rubric through lexical descriptors into a LSA semantic space transforming these LSA space in such a way that the first dimensions captures the meaning of the rubric; b) to compare these two automatic assessment methods into two expository texts that differ in complexity (complex/easy), length (1,300/500 words), different readers (University/High School students), and quality of summaries following the criteria from human graders. To validate Inbuilt Rubric, four versions of the rubric (that is, the combination of few/many lexical descriptors and weighted/not weighted by abstract dimensions) were elaborated and were analyzed according to the quality of the evaluated summaries.…”