Artificial Intelligence–Based Co-Facilitator (AICF) for Detecting and Monitoring Group Cohesion Outcomes in Web-Based Cancer Support Groups: Single-Arm Trial Study

Abstract

Background: Commonly offered as supportive care, therapist-led online support groups (OSGs) are a cost-effective way to provide support to individuals affected by cancer. One important indicator of a successful OSG session is group cohesion; however, monitoring group cohesion can be challenging due to the lack of non-verbal cues and in-person interactions in textbased OSGs. The Artificial Intelligence-based Co-Facilitator (AICF) was designed to contextually identify therapeutic outcomes from conversations and produce real-time analytics. Objective:The aim of this study was to develop a method to train and evaluate AICF's capacity to monitor group cohesion.Methods: Human scorers used a confusion matrix to evaluate the performance of AICF. AICF performance was also compared against the natural language processing software Linguistic Inquiry Word Count (LIWC).Results: AICF was trained on 80,000 messages obtained from Cancer Chat Canada. We tested AICF on 34,048 messages. Human experts scored 6,797 (20%) of the messages to evaluate the ability of AICF to classify group cohesion. Results showed that machine-learning algorithms combined with human input can detect group cohesion, a clinically meaningful indicator of effective OSGs. After re-training with human input, AICF reached a F1-score of 0.82. AICF performed slightly better at identifying group cohesion compared to LIWC.Conclusions: AICF has the potential to assist therapists by detecting discord in the group amenable to real-time intervention. Overall, AICF presents a unique opportunity to strengthen patient-centered care in virtual settings by attending to individual needs.