Efficacy of ChatGPT in Cantonese Sentiment Analysis: A Comparative Study (Preprint)

Abstract

Background Sentiment analysis is a significant yet difficult task in natural language processing. The linguistic peculiarities of Cantonese, including its high similarity with Standard Chinese, its grammatical and lexical uniqueness, and its colloquialism and multilingualism, make it different from other languages and pose additional challenges to sentiment analysis. Recent advances in models such as ChatGPT offer potential viable solutions. Objective This study investigated the efficacy of GPT-3.5 and GPT-4 in Cantonese sentiment analysis in the context of web-based counseling and compared their performance with other mainstream methods, including lexicon-based methods and machine learning approaches. Methods We analyzed transcripts from a web-based, text-based counseling service in Hong Kong, including a total of 131 individual counseling sessions and 6169 messages between counselors and help-seekers. First, a codebook was developed for human annotation. A simple prompt (“Is the sentiment of this Cantonese text positive, neutral, or negative? Respond with the sentiment label only.”) was then given to GPT-3.5 and GPT-4 to label each message’s sentiment. GPT-3.5 and GPT-4’s performance was compared with a lexicon-based method and 3 state-of-the-art models, including linear regression, support vector machines, and long short-term memory neural networks. Results Our findings revealed ChatGPT’s remarkable accuracy in sentiment classification, with GPT-3.5 and GPT-4, respectively, achieving 92.1% (5682/6169) and 95.3% (5880/6169) accuracy in identifying positive, neutral, and negative sentiment, thereby outperforming the traditional lexicon-based method, which had an accuracy of 37.2% (2295/6169), and the 3 machine learning models, which had accuracies ranging from 66% (4072/6169) to 70.9% (4374/6169). Conclusions Among many text analysis techniques, ChatGPT demonstrates superior accuracy and emerges as a promising tool for Cantonese sentiment analysis. This study also highlights ChatGPT’s applicability in real-world scenarios, such as monitoring the quality of text-based counseling services and detecting message-level sentiments in vivo. The insights derived from this study pave the way for further exploration into the capabilities of ChatGPT in the context of underresourced languages and specialized domains like psychotherapy and natural language processing.

Using Large Language Models for sentiment analysis of health-related social media data: empirical evaluation and practical tips

“…With no annotated data provided, ChatGPT achieved an F-measure of 0.8668, outperforming many commonly used off-the-shelf tools including VADER and the Linguistic Inquiry and Word Count (LIWC) 11 . Fu et al, evaluated ChatGPT on patient-generated texts in Cantonese through online counseling sessions and found that the zero-shot LLM can identify sentiments with over 90% accuracy, outperforming multiple machine learning models developed with annotated datasets 12 . Deiner et al evaluated GPTs’ capacity in rating the probabilities of tweets that indicate conjunctive outbreaks and found that GPT-4 had satisfying agreements with two human raters with Pearson correlation scores of 0.73 and 0.77 13 .…”

Research on the Application of ChatGPT in the Interdisciplinars of Higher Education

“…This is an inevitable choice to adapt to technological trends, meet job market demands, respond to national educational policies, enhance international competitiveness, and promote scientific and technological innovation. The rise of interdisciplinary studies not only helps improve students' employability and innovation skills but is also significant for regional and national economic and technological development [7]- [8]. The integration of ChatGPT into the higher education system of Shandong Province, particularly in interdisciplinary teaching and research, signifies a major transformation in educational models and research methods [9]- [13].…”