Do Artificial Intelligence Chatbots Have a Writing Style? An Investigation into the Stylistic Features of ChatGPT-4

Abstract

Even though Turnitin generates AI (Artificial Intelligence) writing detection reports, these AI reports shall not be used for punitive purposes as Turnitin AI reports accuracy is way below the 98% claimed by Turnitin, as revealed in this study. To assist professors, teachers, and content evaluation stakeholders in their strive to identify AI-generated material, this study examines the stylistic features of case study, business correspondence, and academic writing ChatGPT-4 generated responses by exploring sentence length, paragraph structure, word choice, mood, tense, voice, pronouns, keywords density, lexical density, lexical diversity, and reading ease. The study revealed that ChatGPT-4 case study generated responses are produced in paragraphs of 2 to 3 sentences of 16 to 18 words each. The sentences are mainly formed in imperative mood. The use of the second-person pronoun ‘you’ and the second-person possessive determiner ‘your’ is prevalent. Keywords and lexical density are relatively low, lexical diversity is average, and the reading ease is relatively high. The study also found that ChatGPT-4 business correspondence responses are generated in paragraphs of 2 to 3 sentences of 16 to 20 words each. The sentences are mainly generated in declarative mood thru simple present tense in active voice using third-person singular pronouns. Technical words and abbreviations are used without outlining what they stand for. The keywords density, lexical density, and lexical diversity are high and the reading ease is low. The study also revealed that ChatGPT-4 academic writing generated responses are provided in paragraphs of 3 to 4 sentences of 16 to 19 words each. The sentences are mainly generated in declarative mood using active voice, agentless passive in times, with diverse present tenses. Keywords and lexical densities are high and the lexical diversity is low, which makes the reading ease average difficulty, except for the undefined abbreviations. Noticeably, ChatGPT-4 supports the transgender movement by intentionally using the third-person plural pronoun ‘they’ to refer to a singular.

Comparison of the audiological knowledge of three chatbots – ChatGPT, Bing Chat, and Bard

“…This means that each chatbot generated responses according to a predetermined pattern using more or less the same number of words. There also seemed to be some standard word patterns [24,25]. Perhaps after further use it might be possible to recognize which response came from which chatbot.…”

The Advantages and Disadvantages of ChatGPT in Humanities, Social Sciences and STEM Education: From a Comparative Perspective

“…There are also a few moral issues with ChatGPT: which brings up issues with authorship, responsibility, and transparency [9]. Turnitin, a plagiarism detection tool, initially showed low similarity scores for ChatGPT-4 responses, but adding human-written paragraphs increased the AI index to 100% [10].…”

ESD goals and soft skills competencies through constructivist approaches to teaching: an integrative review

“…Students regardless of their geographical location have access to a big number of educational technologies and networks. The challenge in this era, 22 years after, is making sure that students use technology and artificial intelligence software in an ethical manner that does not affect their learning development [21]− [23]. In addition, the use of material shall be context-based.…”