How Trustworthy is ChatGPT? The Case of Bibliometric Analyses

Abstract

The introduction of the AI-powered chatbot ChatGPT by OpenAI has sparked much interest and debate among academic researchers. Commentators from different scientific disciplines have raised many concerns and issues, especially related to the ethics of using these tools in scientific writing and publications. In addition, there has been discussions about whether ChatGPT is trustworthy, effective, and useful in increasing researchers’ productivity. Therefore, in this paper, we evaluate ChatGPT’s performance on tasks related to bibliometric analysis, by comparing the output provided by the chatbot with a recently conducted bibliometric study on the same topic. The findings show that there are large discrepancies and ChatGPT’s trustworthiness is low in this particular area. Therefore, researchers should exercise caution when using ChatGPT as a tool in bibliometric studies.

Network Visualization of ChatGPT Research: A Study Based on Term and Keyword Co-occurrence Network Analysis

“…These network visualization tools can be helpful in analyzing the usage and impact of ChatGPT's keywords in academic publications. Recently two research papers have been carried out on ChatGPT and bibliometrics (Kirtania, 2023;Farhat, Sohail & Madsen, 2023) and here ChatGPT is discussed with advantages, limitations and trustworthiness. Visualizing this type of networks for analyzing the impact of ChatGPT's research and its keywords in the academic community, and can help researchers gain valuable insights into the broader research landscape.…”