Assessing the reliability of ChatGPT: a content analysis of self-generated and self-answered questions on clear aligners, TADs and digital imaging

Abstract

Introduction: Artificial Intelligence (AI) is a tool that is already part of our reality, and this is an opportunity to understand how it can be useful in interacting with patients and providing valuable information about orthodontics. Objective: This study evaluated the accuracy of ChatGPT in providing accurate and quality information to answer questions on Clear aligners, Temporary anchorage devices and Digital imaging in orthodontics. Methods: forty-five questions and answers were generated by the ChatGPT 4.0, and analyzed separately by five orthodontists. The evaluators independently rated the quality of information provided on a Likert scale, in which higher scores indicated greater quality of information (1 = very poor; 2 = poor; 3 = acceptable; 4 = good; 5 = very good). The Kruskal-Wallis H test (p< 0.05) and post-hoc pairwise comparisons with the Bonferroni correction were performed. Results: From the 225 evaluations of the five different evaluators, 11 (4.9%) were considered as very poor, 4 (1.8%) as poor, and 15 (6.7%) as acceptable. The majority were considered as good [34 (15,1%)] and very good [161 (71.6%)]. Regarding evaluators’ scores, a slight agreement was perceived, with Fleiss’s Kappa equal to 0.004. Conclusions: ChatGPT has proven effective in providing quality answers related to clear aligners, temporary anchorage devices, and digital imaging within the context of interest of orthodontics.

Evaluating Chat Generative Pre-trained Transformer Responses to Common Pediatric In-toeing Questions

“…Our findings reveal that ChatGPT demonstrates a high level of performance, with 90% of responses requiring minimal to no clarification. Studies evaluating ChatGPT’s accuracy in responding to common patient questions show similar rates of satisfactory responses ranging from 60% to 100% 5,6,21–26 . These results underscore the potential of AI technologies like ChatGPT to support health care professionals in delivering timely and accurate information to patients and caregivers.…”

Accuracy and Completeness of ChatGPT-Generated Information on Interceptive Orthodontics: A Multicenter Collaborative Study

“…Tanaka et al, have reported that ChatGPT has proven effective in providing quality answers related to clear aligners, temporary anchorage devices, and digital imaging within the context of interest of orthodontics [34]. Tanaka et al's study and the present study are very similar, but in the present study, a group of ten orthodontists with a postgraduate degree from ten different Italian orthodontics postgraduate schools gave their evaluation, while Tanaka et al's research group was formed of five general orthodontists.…”

Large Language Models take on the AAMC Situational Judgment Test: Evaluating Dilemma-Based Scenarios

“…In using ChatGPT-3.5, ChatGPT-4.0, and Bard to rate the effectiveness of each solution, a Fleiss Kappa was utilized to assess inter-model agreement. This study serves as the rst instance in using LLMs as graders through a Fleiss Kappa; the purpose of this methodology is to determine if there are tendencies across the models in relation to their answer choice (Draschl, 2023;O'Hagan, 2023;Schopow, 2023;Tanaka, 2023;Biswas, 2023;Gorelik, 2023;Walker, 2023). Fair agreement was perceived among the three LLMs on raw accuracy and adjusted accuracy, indicating that despite ChatGPT-4.0's strong performance, the models tend to agree in correctly determining effectiveness.…”