The accuracy of artificial intelligence–based virtual assistants in responding to routinely asked questions about orthodontics

Abstract

Objectives To evaluate the utility and efficiency of four voice-activated, artificial intelligence–based virtual assistants (Alexa, Google Assistant, Siri, and Cortana) in addressing commonly asked patient questions in orthodontic offices. Materials and Methods Two orthodontists, an orthodontic resident, an oral and maxillofacial radiologist, and a dental student used a standardized list of 12 questions to query and evaluate the four most common commercial virtual assistant devices. A modified Likert scale was used to evaluate their performance. Results Google Assistant had the lowest (best) mean score, followed by Siri, Alexa, and Cortana. The score of Google Assistant was significantly lower than Alexa and Cortana. There was significant variablity in virtual assistant response scores among the evaluators, with the exception of Amazon Alexa. Lower scores indicated superior efficiency and utility. Conclusions The common commercially available virtual assistants tested in this study showed significant differences in how they responded to users. There were also significant differences in their performance when responding to common orthodontic queries. An intelligent virtual assistant with evidence-based responses specifically curated for orthodontics may be a good solution to address this issue. The investigators in this study agreed that such a device would provide value to patients and clinicians.

Comparative Analysis of Artificial Intelligence Virtual Assistant and Large Language Models in Post-Operative Care

“…When comparing various virtual assistants including Siri, Alexa, Google Assistant, and Cortana in their ability to respond to orthodontic-related patient inquiries, Google Assistant emerged as the most effective, achieving the highest mean score on a modified Likert scale [13]. However, the variance in efficacy across these platforms highlights the necessity for more specialized training in addressing patient questions within this specific domain.…”

“…Notably, Google Assistant predominantly sources its content from private practice websites, and these tools are generally not designed with a primary focus on patient query resolution, but rather on providing information across various topics. Moreover, the study that provided these insights had a smaller sample size compared to ours and limited its evaluation to the relevance of responses, omitting other critical dimensions of patient education that can be significant [13]. In a comparative study [33], the accuracy of ChatGPT-4, BARD, and YOU.com was evaluated in determining the stage and management of tonsillar cancer based on a patient's clinical vignette.…”

“…By employing natural language processing (NLP) techniques, these AI-powered virtual assistants (AIVA) can comprehend conversations and respond appropriately, utilizing their training data to execute specific tasks effectively [12]. These systems are envisioned as alternatives to healthcare professionals for performing routine patient education tasks both efficiently and cost-effectively, thereby reducing the workload of healthcare workers [13][14][15]. To further assess their efficacy, particularly in answering patient queries post-surgery, we aimed to compare these AI tools with advanced large language models (LLMs) such as Google's BARD and OpenAI's Generative Pretrained Transformer (GPT), which represent the forefront of NLP tools.…”

Use of artificial intelligence in medical education: A strength or an infirmity

“…For example, Virtual Assistants (such as Siri, Alexa, and Google Assistant) are used to understand and respond to users' queries, perform tasks, and provide personalized recommendations. 4 Similarly, Natural Language Processing (NLP) is used for applications such as chatbots, language translation, sentiment analysis, voice recognition, and speech-to-text applications. 5 It enables machines to understand and communicate by generating human language.…”