Comparing letters written by humans and ChatGPT: A preliminary study

Abstract

ObjectiveThere are no criteria for what type of manuscript and to what extent ChatGPT use is permissible in writing manuscripts. I aimed to determine which, human or ChatGPT, writes more readable letters to the editor and whether ChatGPT writes letters mimicking a certain person. I aimed to provide hints as to what makes the difference between humans and ChatGPT.MethodsThis is a descriptive pilot study. I tasked ChatGPT (version 3.5) with generating a disagreement letter to my previous article (Letter 0). I wrote a letter involving three weaknesses of the addressed article (Letter 1). I provided ChatGPT with these three weaknesses and tasked it with generating a letter (Letter 2). Then, I supplied my authored letters and tasked ChatGPT with emulating my writing style (Letter 3). Eight professors evaluated the letters' readability and ChatGPT assessed which letter was more likely to be accepted.ResultsChatGPT produced coherent letters (Letters 0 and 2). Professors rated the readability of Letters 1 and 2 similarly, finding Letter 3 less readable. ChatGPT determined that the human‐authored Letter 1 had a slightly higher acceptance likelihood than the ChatGPT‐generated Letter 2. Although ChatGPT identified personal writing styles, its mimicry did not enhance the letter's quality.ConclusionThis preliminary experiment indicates that human‐written letters are perceived to be as readable as, or no less readable than, ChatGPT‐generated ones. It suggests that human touch, with its inherent enthusiasm, is essential for effective letter writing. Further comprehensive investigations are warranted to ascertain the extent to which ChatGPT can be used in this domain.