Towards Automatic Narrative Coherence Prediction

Abstract

Research in Psychology has shown that stories people tell about themselves, and how they recall their experiences, reveal a lot about their individual characteristics and mental well-being. The Narrative Coherence Coding Scheme (NaCCS) is a set of guidelines established in psychology research for annotating the "coherence" of a narrative along three dimensions: context, chronology and theme. A significant correlation was found between a narrative's coherence score and independently collected mental health markers of the narrator. Currently, all coherence annotations are done manually; a time consuming task which drains vital resources. In this paper, we propose an Artificial Intelligence based approach involving Natural Language Processing (NLP) to predict a narrative's coherence score (4-class classification problem). We explore a number of techniques, ranging from traditional machine learning models such as Support Vector Machines (SVM) to pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers). BERT produced the best results for all dimensions in terms of accuracy: 53.7% (context), 71.8% (chronology), and 69.6% (theme). The location of information in the narratives (beginning, end, throughout) was helpful in improving predictions.

Transformer based Model for Coherence Evaluation of Scientific Abstracts: Second Fine-tuned BERT

“…The authors agree with the work of Bendevski et al [30] when he mentions that BERT offers better results for the evaluation of coherence than traditional machine learning methods, since the latter cannot understand the context of a text as does BERT, in addition to that BERT offers greater scalability and guarantees the Transfer learning process.…”

“…In the work of Bendevski et al (2021), a comparative analysis of different artificial intelligence methods for predicting the coherence score of narrative documents was proposed, where it was evidenced that BERT produces better results compared to traditional machine learning methods such as: Linear Regression, Support Vector Machine, Random Forest. They establish dimensions for coherence which are: Context, Chronology and Theme, each of these dimensions possess narrative texts with a coherence score of 0-3 (4-class classification) [30].…”

Automatic Context-Aware Inference of Engagement in HMI: A Survey

“…The presence of certain pathologies can alter the way an individual behaves. For example, pathologies that have an effect on social behaviour include Autism Spectrum Disorder (ASD), Attention Deficit Hyperactivity Disorder (ADHD), Major Depression Disorder (MDD), Bipolar Disorder (BP), among others [91], [92], [93]. For instance, individuals with ASD are characterized by deficiencies in demonstrating proper social cues in social interaction contexts [94].…”