Towards a Reliable and Rapid Automated Grading System in Facial Palsy Patients: Facial Palsy Surgery Meets Computer Science

Knoedler, Leonard; Baecher, Helena; Kauke, Martin; Prantl, Lukas; Machens, Hans‐Günther; Scheuermann, Philipp; Palm, Christoph; Baumann, Raphaël; Kehrer, Andreas; Knoedler, Samuel

doi:10.3390/jcm11174998

Cited by 16 publications

(12 citation statements)

References 75 publications

(86 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Artificial intelligence (AI)-supported tools have been proposed for a variety of medical scenarios, including preoperative outcome simulation, patient education, and automated disease grading [6][7][8][9]. Recently, chatbots such as ChatGPT have emerged as next-generation AI technology.…”

Section: Introductionmentioning

confidence: 99%

Pure Wisdom or Potemkin Villages? A Comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 Style Questions: Quantitative Analysis

Knoedler,

Alfertshofer,

Knoedler

et al. 2024

JMIR Med Educ

Self Cite

View full text Add to dashboard Cite

Background The United States Medical Licensing Examination (USMLE) has been critical in medical education since 1992, testing various aspects of a medical student’s knowledge and skills through different steps, based on their training level. Artificial intelligence (AI) tools, including chatbots like ChatGPT, are emerging technologies with potential applications in medicine. However, comprehensive studies analyzing ChatGPT’s performance on USMLE Step 3 in large-scale scenarios and comparing different versions of ChatGPT are limited. Objective This paper aimed to analyze ChatGPT’s performance on USMLE Step 3 practice test questions to better elucidate the strengths and weaknesses of AI use in medical education and deduce evidence-based strategies to counteract AI cheating. Methods A total of 2069 USMLE Step 3 practice questions were extracted from the AMBOSS study platform. After including 229 image-based questions, a total of 1840 text-based questions were further categorized and entered into ChatGPT 3.5, while a subset of 229 questions were entered into ChatGPT 4. Responses were recorded, and the accuracy of ChatGPT answers as well as its performance in different test question categories and for different difficulty levels were compared between both versions. Results Overall, ChatGPT 4 demonstrated a statistically significant superior performance compared to ChatGPT 3.5, achieving an accuracy of 84.7% (194/229) and 56.9% (1047/1840), respectively. A noteworthy correlation was observed between the length of test questions and the performance of ChatGPT 3.5 (ρ=–0.069; P=.003), which was absent in ChatGPT 4 (P=.87). Additionally, the difficulty of test questions, as categorized by AMBOSS hammer ratings, showed a statistically significant correlation with performance for both ChatGPT versions, with ρ=–0.289 for ChatGPT 3.5 and ρ=–0.344 for ChatGPT 4. ChatGPT 4 surpassed ChatGPT 3.5 in all levels of test question difficulty, except for the 2 highest difficulty tiers (4 and 5 hammers), where statistical significance was not reached. Conclusions In this study, ChatGPT 4 demonstrated remarkable proficiency in taking the USMLE Step 3, with an accuracy rate of 84.7% (194/229), outshining ChatGPT 3.5 with an accuracy rate of 56.9% (1047/1840). Although ChatGPT 4 performed exceptionally, it encountered difficulties in questions requiring the application of theoretical concepts, particularly in cardiology and neurology. These insights are pivotal for the development of examination strategies that are resilient to AI and underline the promising role of AI in the realm of medical education and diagnostics.

show abstract

Section: Introductionmentioning

confidence: 99%

Pure Wisdom or Potemkin Villages? A Comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 Style Questions: Quantitative Analysis

Knoedler,

Alfertshofer,

Knoedler

et al. 2024

JMIR Med Educ

Self Cite

View full text Add to dashboard Cite

show abstract

“…Compared with other deep learning models that leverage on qualitative grading system to evaluate facial pathologies ( [59,60]), our LSTM model provides an objective assessment that exploits motion capture data. Furthermore, marker positions are more accurate than landmark positions because metric measurements are directly recorded rather than being estimated from the image space ( [22]).…”

Section: Discussionmentioning

confidence: 99%

DeepSmile: Anomaly Detection Software for Facial Movement Assessment

et al. 2023

View full text Add to dashboard Cite

Facial movements are crucial for human interaction because they provide relevant information on verbal and non-verbal communication and social interactions. From a clinical point of view, the analysis of facial movements is important for diagnosis, follow-up, drug therapy, and surgical treatment. Current methods of assessing facial palsy are either (i) objective but inaccurate, (ii) subjective and, thus, depending on the clinician’s level of experience, or (iii) based on static data. To address the aforementioned problems, we implemented a deep learning algorithm to assess facial movements during smiling. Such a model was trained on a dataset that contains healthy smiles only following an anomaly detection strategy. Generally speaking, the degree of anomaly is computed by comparing the model’s suggested healthy smile with the person’s actual smile. The experimentation showed that the model successfully computed a high degree of anomaly when assessing the patients’ smiles. Furthermore, a graphical user interface was developed to test its practical usage in a clinical routine. In conclusion, we present a deep learning model, implemented on open-source software, designed to help clinicians to assess facial movements.

show abstract

“…Other studies used different techniques such as facial measurements, facial animation units, and machine learning algorithms to quantify facial palsy. [14][15][16] While these alternative methods demonstrate different ways to analyze facial palsy, the evaluation remains static and unable to assess real-time dynamic function.…”

Section: Artificial Intelligence-driven Solutionmentioning

confidence: 99%

Deep Learning for the Assessment of Facial Nerve Palsy: Opportunities and Challenges

et al. 2023

View full text Add to dashboard Cite

Automated evaluation of facial palsy using machine learning offers a promising solution to the limitations of current assessment methods, which can be time-consuming, labor-intensive, and subject to clinician bias. Deep learning-driven systems have the potential to rapidly triage patients with varying levels of palsy severity and accurately track recovery over time. However, developing a clinically usable tool faces several challenges, such as data quality, inherent biases in machine learning algorithms, and explainability of decision-making processes. The development of the eFACE scale and its associated software has improved clinician scoring of facial palsy. Additionally, Emotrics is a semiautomated tool that provides quantitative data of facial landmarks on patient photographs. The ideal artificial intelligence (AI)-enabled system would analyze patient videos in real time, extracting anatomic landmark data to quantify symmetry and movement, and estimate clinical eFACE scores. This would not replace clinician eFACE scoring but would offer a rapid automated estimate of both anatomic data, similar to Emotrics, and clinical severity, similar to the eFACE. This review explores the current state of facial palsy assessment, recent advancements in AI, and the opportunities and challenges in developing an AI-driven solution.

show abstract

Towards a Reliable and Rapid Automated Grading System in Facial Palsy Patients: Facial Palsy Surgery Meets Computer Science

Cited by 16 publications

References 75 publications

Pure Wisdom or Potemkin Villages? A Comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 Style Questions: Quantitative Analysis

Pure Wisdom or Potemkin Villages? A Comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 Style Questions: Quantitative Analysis

DeepSmile: Anomaly Detection Software for Facial Movement Assessment

Deep Learning for the Assessment of Facial Nerve Palsy: Opportunities and Challenges

Contact Info

Product

Resources

About