Surviving ChatGPT in healthcare

Liu, Zhengliang; Zhang, Lu; Wu, Zihao; Yu, Xiaowei; Cao, Chao; Dai, Haixing; Liu, Ninghao; Liu, Jun; Liu, Wei; Li, Quanzheng; Shen, Dinggang; Li, Xiang; Zhu, Dajiang; Liu, Tianming

doi:10.3389/fradi.2023.1224682

Cited by 2 publications

(1 citation statement)

References 52 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Students and trainees could use ChatGPT for assignments and exams. In addition, using such tools can hinder the students' learning process, especially at the current stage, where curriculum design has not been updated accordingly [51]. Finally, as more patients rely on internet searches to seek medical advice, it is important to mark the AIGC, especially that related to medicine, with "Generated by AIGC" labels.…”

Section: Principal Resultsmentioning

confidence: 99%

Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study

Liao,

Liu,

Dai

et al. 2023

JMIR Med Educ

Self Cite

View full text Add to dashboard Cite

Background Large language models, such as ChatGPT, are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the internet. However, medical texts, such as clinical notes and diagnoses, require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to health care and the general public. Objective This study is among the first on responsible artificial intelligence–generated content in medicine. We focus on analyzing the differences between medical texts written by human experts and those generated by ChatGPT and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods We first constructed a suite of data sets containing medical texts written by human experts and generated by ChatGPT. We analyzed the linguistic features of these 2 types of content and uncovered differences in vocabulary, parts-of-speech, dependency, sentiment, perplexity, and other aspects. Finally, we designed and implemented machine learning methods to detect medical text generated by ChatGPT. The data and code used in this paper are published on GitHub. Results Medical texts written by humans were more concrete, more diverse, and typically contained more useful information, while medical texts generated by ChatGPT paid more attention to fluency and logic and usually expressed general terminologies rather than effective information specific to the context of the problem. A bidirectional encoder representations from transformers–based model effectively detected medical texts generated by ChatGPT, and the F1 score exceeded 95%. Conclusions Although text generated by ChatGPT is grammatically perfect and human-like, the linguistic characteristics of generated medical texts were different from those written by human experts. Medical text generated by ChatGPT could be effectively detected by the proposed machine learning algorithms. This study provides a pathway toward trustworthy and accountable use of large language models in medicine.

show abstract

Section: Principal Resultsmentioning

confidence: 99%