A Comparison of ChatGPT and Fine-Tuned Open Pre-Trained Transformers (OPT) Against Widely Used Sentiment Analysis Tools: Sentiment Analysis of COVID-19 Survey Data

Abstract

Background Health care providers and health-related researchers face significant challenges when applying sentiment analysis tools to health-related free-text survey data. Most state-of-the-art applications were developed in domains such as social media, and their performance in the health care context remains relatively unknown. Moreover, existing studies indicate that these tools often lack accuracy and produce inconsistent results. Objective This study aims to address the lack of comparative analysis on sentiment analysis tools applied to health-related free-text survey data in the context of COVID-19. The objective was to automatically predict sentence sentiment for 2 independent COVID-19 survey data sets from the National Institutes of Health and Stanford University. Methods Gold standard labels were created for a subset of each data set using a panel of human raters. We compared 8 state-of-the-art sentiment analysis tools on both data sets to evaluate variability and disagreement across tools. In addition, few-shot learning was explored by fine-tuning Open Pre-Trained Transformers (OPT; a large language model [LLM] with publicly available weights) using a small annotated subset and zero-shot learning using ChatGPT (an LLM without available weights). Results The comparison of sentiment analysis tools revealed high variability and disagreement across the evaluated tools when applied to health-related survey data. OPT and ChatGPT demonstrated superior performance, outperforming all other sentiment analysis tools. Moreover, ChatGPT outperformed OPT, exhibited higher accuracy by 6% and higher F-measure by 4% to 7%. Conclusions This study demonstrates the effectiveness of LLMs, particularly the few-shot learning and zero-shot learning approaches, in the sentiment analysis of health-related survey data. These results have implications for saving human labor and improving efficiency in sentiment analysis tasks, contributing to advancements in the field of automated sentiment analysis.

Sentiment Analysis with Graph Convolutional Networks using directed PMI

“…[Rezaeinia et al(2019)Rezaeinia, Rahmani, Ghodsi, and Veisi] improved popular Word2Vec and GloVe word embeddings by Part-of-Speech (POS) tagging techniques, lexicon-based approaches, and word position algorithms. [Yu et al(2017)Yu, Wang, Lai, and Zhang] also designed a refinement model to enhance any pre-trained word vector in a way that both semantic and sentiment are considered while setting the distance.…”

“…and [Lotfi et al(2021)Lotfi, Mirzarezaee, Hosseinzadeh, and Seydi]. But most of them can't be applied to directed graphs.…”

Using Large Language Models for sentiment analysis of health-related social media data: empirical evaluation and practical tips

“…For example, Lossio-Ventura et al evaluated zero-shot ChatGPT for analyzing the sentiments conveyed through COVID-19 free-text survey responses. With no annotated data provided, ChatGPT achieved an F-measure of 0.8668, outperforming many commonly used off-the-shelf tools including VADER and the Linguistic Inquiry and Word Count (LIWC) 11 . Fu et al, evaluated ChatGPT on patient-generated texts in Cantonese through online counseling sessions and found that the zero-shot LLM can identify sentiments with over 90% accuracy, outperforming multiple machine learning models developed with annotated datasets 12 .…”

Pediatric Cancer Communication on Twitter: Natural Language Processing and Qualitative Content Analysis

“…Additionally, the launch of Open AI’s chatbot, ChatGPT, provides a novel tool for nonlexicon-based sentiment analysis via text-based chat inquiries. Emerging research suggests that ChatGPT demonstrates superior performance in sentiment analysis of free-text responses [ 24 ].…”