Supporting Cognitive and Emotional Empathic Writing of Students

Abstract

We present an annotation approach to capturing emotional and cognitive empathy in student-written peer reviews on business models in German. We propose an annotation scheme that allows us to model emotional and cognitive empathy scores based on three types of review components. Also, we conducted an annotation study with three annotators based on 92 student essays to evaluate our annotation scheme. The obtained inter-rater agreement of α=0.79 for the components and the multi-π=0.41 for the empathy scores indicate that the proposed annotation scheme successfully guides annotators to a substantial to moderate agreement. Moreover, we trained predictive models to detect the annotated empathy structures and embedded them in an adaptive writing support system for students to receive individual empathy feedback independent of an instructor, time, and location. We evaluated our tool in a peer learning exercise with 58 students and found promising results for perceived empathy skill learning, perceived feedback accuracy, and intention to use. Finally, we present our freely available corpus of 500 empathy-annotated, student-written peer reviews on business models and our annotation guidelines to encourage future research on the design and development of empathy support systems.

Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support

“…Though our human evaluation captures platform users' perceptions of empathy in responses, it is unlikely to measure empathy from the perspective of psychology theory given the limited training of TalkLife users. Therefore, we conducted a second complementary evaluation by applying the theory-based empathy classification model proposed by Sharma et al 29 , which assigns a score between 0 and 6 to each response and has been validated and used in prior work 47,[74][75][76] . Note that this approach evaluates empathy expressed in responses and not the empathy perceived by support seekers of the original seeker post (Discussion).…”

Designing Conversational Evaluation Tools

“…Hence, further studies are needed to confirm our results with more precise measurements. With ML algorithms advancing and also measuring more sophisticated constructs of human communication such as empathy (e.g., [82]) or argumentation (e.g., [80]), we believe more precise methods to measure the Gricean Maxims might evolve in future work that can be used to assess response quality more accurately [24]. In addition, future research could investigate more advanced metrics on response quality.…”

A Design Space for Intelligent and Interactive Writing Assistants

“…Empathy focuses on the user's ability to emotionally and cognitively empathize with others in the writing process. This empathetic focus was observed within educational contexts, where students are instructed to write more empathetic peer reviews [251,252]. Cognition looks at cognitive aspects like the user's focus, sense of immersion, and cognitive load, as systems can increase cognitive engagement by tackling phenomena like writer's block [12,45,220].…”

“…Understanding age is important when tailoring systems to accommodate the developmental and cognitive characteristics of users across a wide age spectrum, ranging from young, pre-literate children [270] to adolescents [89,216]. Similarly, education level and background of users, whether they are high school students [216] or university graduates [2,251], indicates varying cognitive and learning competencies, which in turn may influence how users interact with and critically engage in systems. Finally, some studies emphasize tailoring system designs to a specific profession, such as professional creative writers [80,174], ensuring that the systems meet the unique writing needs of different professions.…”