Towards Evaluating Narrative Quality In Student Writing

Abstract

This work lays the foundation for automated assessments of narrative quality in student writing. We first manually score essays for narrative-relevant traits and sub-traits, and measure inter-annotator agreement. We then explore linguistic features that are indicative of good narrative writing and use them to build an automated scoring system. Experiments show that our features are more effective in scoring specific aspects of narrative quality than a state-of-the-art feature set.

Proceedings of the Second Workshop on Storytelling

“…Total score (range 0-11) is the sum of Organization, Development and Conventions. Not surprisingly, the Organization, Development, Narrative and Total scores are highly intercorrelated (0.88 and higher, see Table 3 in Somasundaran et al (2018)). For the present study, we used the Narrative scores, focusing on essay narrative quality and de-emphasizing grammar and mechanics.…”

“…We used a corpus of narrative essays 4 provided by Somasundaran et al (2018). The corpus consists of 940 narrative essays written by school students from grade levels 7-12.…”

“…All essays were manually scored by experienced research assistants (Somasundaran et al, 2018), using a rubric that was created by education experts and teachers, and presented by Smarter Balanced assessment consortium, an assessment aligned to U.S. State Standards for grades K-12 ( Smarter Balanced, 2014b,a).…”

“…Scores are on a scale of 0-3 integer points (3 is the highest score). Somasundaran et al (2018) computed Narrative and Total composite scores for each essay. The Narrative score (range 0-8) is the sum of Organization and Development scores.…”

An exploration of automated narrative analysis via machine learning

“…Though this is a useful tool for many contexts, it provides only a handful of measures that are directed towards narrative analysis, with a greater focus on text cohesion [24]. In 2018, researchers working under ETS came close to creating a clinically relevant automated narrative sample analysis system [25]. This work serves as an important proof of concept, however it still leaves a number of gaps in the literature.…”