Using latent semantic analysis to grade brief summaries: some proposals

Abstract

In this paper, we present several proposals in order to improve the LSA tools to evaluate brief summaries (less than 50 words) of narrative and expository texts. First, we analyse the quality of six different methods assessing essays that have been widely employed before (Foltz et al., 2000). The second objective is to analyse how new algorithms inspired by some authors (Denhière et al., 2007) that try to emulate human behaviour to improve the reliability of LSA with human graders when assessing short summaries, compared with standard LSA use in expository text. Finally, we present an assessment method to combine LSA as a semantic computational linguistic model with ROUGE-N as a lexical model, to show how combining different automatic evaluation systems (LSA and ROUGE) can improve the quality of assessments in different academic levels.

Computerized summary scoring: crowdsourcing-based latent semantic analysis

“…The units may be words, phrases, clauses, sentences, paragraphs (Foltz, 1996;Landauer, 1998;Landauer & Dumais, 1997;Landauer, Foltz, & Laham, 1998), or summaries (Foltz, 1996;E. Kintsch et al, 2000;Olmos, León, Escudero, & Jorge-Botana, 2011;Olmos, León, Jorge-Botana, & Escudero, 2009. LSA uses the geometric cosine between two vectors to compute the conceptual similarity between any two units of the text.…”

“…Empirical evidence has shown that LSA robustly predicts the semantic quality of summaries as perceived by humans (Jorge-Botana, Luzón, Gómez-Veiga, & Martín-Cordero, 2015;E. Kintsch et al, 2000;Olmos et al, 2011Olmos et al, , 2013Sung, Liao, Chang, Chen, & Chang, 2016;Wade-Stein & Kintsch, 2004). One successful system that uses LSA to automatically assess summaries is Summary Street® (E. Kintsch et al, 2000;Wade-Stein & Kintsch, 2004).…”

“…To address these limitations, some researchers developed new LSA algorithms in order to improve the reliability of LSA to capture the meaning of short summaries (25-50 words). Olmos et al (2009) and Olmos et al (2011) compared three new algorithms: a semantic common network algorithm (W. Kintsch, 2001Kintsch, , 2002, a best-dimension reduction measure of the latent semantic space (Hu, Cai, Wiemer-Hastings, Graesser, & McNamara, 2007), and the Euclidean distance measure (Rehder et al, 1998). The results showed that a bestdimension algorithm best predicted the short expository summaries, as compared to other LSA algorithms.…”

“…The holistic method showed higher accuracy for the overall quality of a summary, whereas the analytic method had the advantage of providing more specific details in terms of the sublevel components (Foltz, Gilliam, & Kendall, 2000). Previous research compared five holistic methods (crowdsourcing summary, individual summaries, source text, expert summary, pregraded summaries) and two analytic methods (sentence vs. main sentence) Olmos et al, 2011). The analytic method either calculated the cosine scores between a target summary and each sentence in the original source text or computed the cosine scores between each sentence in a target summary and a set of important sentences that experts had identified in the source text.…”

“…Empirical evidence will be needed to confirm the extent to which crowdsourcing LSA (good, medium, poor, or all crowdsourcing summaries) predicts human summary scores as compared to other LSA methods (good, medium, and poor expert summaries along with source texts), which was the goal of this study. Olmos et al (2011) proposed another alternative LSA method that used students' summaries. This method compared the target summary with each individual student's summary and then the average cosine score was assigned to the target summary.…”