Using Semantic Technologies for Formative Assessment and Scoring in Large Courses and MOOCs

Abstract

Formative assessment and personalised feedback are commonly recognised as key factors both for improving students' performance and increasing their motivation and engagement (Gibbs and Simpson, 2005). Currently, in large and massive open online courses (MOOCs), technological solutions to give feedback are often limited to quizzes of different kinds. At present, one of our challenges is to provide feedback for open-ended questions through semantic technologies in a sustainable way.To face such a challenge, our academic team decided to use a test based on latent semantic analysis (LSA) and chose an automatic assessment tool named G-Rubric. G-Rubric was developed by researchers at the Developmental and Educational Psychology Department of UNED (Spanish national distance education university). By using G-Rubric, automated formative and iterative feedback was provided to students for different types of open-ended questions (70-800 words). This feedback allowed students to improve their answers and writing skills, thus contributing both to a better grasp of concepts and to the building of knowledge.In this paper, we present the promising results of our first experiences with UNED business degree students along three academic courses (2014-15, 2015-16 and 2016-17). These experiences show to what extent assessment software such as G-Rubric is mature enough to be used with students. It offers them enriched and personalised feedback that proved entirely satisfactory. Furthermore, G-Rubric could help to deal with the problems related to manual grading, even though our final goal is not to replace tutors by semantic tools, but to give support to tutors who are grading assignments.

Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

“…Agejev andŠnajder (2017) uses ROUGE and BLEU in assessing summary writing from college L2 students. Santamaría Lancho et al (2018) show that using G-Rubric, an LSA-based tool applying rubric assessment, helps the instructors grade the short text answers to open-ended questions, and proves to be reliable, with a Pearson correlation between human graders and G-Rubric of 0.72.…”

Procesos de feedback para fomentar la autorregulación con soporte tecnológico en la educación superior: Revisión sistemática

“…En concreto el VSR puede tener el potencial de ayudar a los estudiantes a reflexionar sobre el feedback. Santamaría et al (2018) realizan una intervención consistente en dar retroalimentación a preguntas abiertas a través de tecnologías semánticas, derivadas del AEA (Evaluación Automática de Ensayos), de manera sostenible basándose en análisis semántico latente (LSA) de preguntas abiertas a partir del G-rubric para apoyar el papel de los tutores on line. El objetivo es automatizar las correcciones de los ensayos para que la fiabilidad tanto entre evaluadores como del mismo evaluador en diversas pruebas aumente.…”

Analytical Techniques for Developing Argumentative Writing in STEM

“…G-Rubric converts LSA vector space with latent dimensions of meaning to a new vector space with semantic grounding i (e.g., 300), a fixed number of relevant concepts. It has been used to give college students iterative feedback during revision of source-based summaries [53], and with business students in a MOOC [55]. Concept maps are another rubric-free feedback method.…”

“…To summarize, studies show automated analysis can support formative assessment during writing instruction by helping the instructor to provide prompt feedback [40], [47], [48], to students while revising their drafts [44], [53], [55], which can lead to improved writing skills [42], [57]. Machine learning methods as used in PEG, C-rater-ML, G-Rubric and [40] generalize better than Coh-Metrix alone, although Coh-Metrix provides useful features for the machine learning approach used in [40].…”

“…Machine learning methods as used in PEG, C-rater-ML, G-Rubric and [40] generalize better than Coh-Metrix alone, although Coh-Metrix provides useful features for the machine learning approach used in [40]. Santamaría Lancho et al [55] suggest that automated support could also be integrated with human grading to improve the consistency and reliability of summative assessment.…”