Superior Performance of Artificial Intelligence Models in English Compared to Arabic in Infectious Disease Queries

Abstract

Background Assessment of artificial intelligence (AI)-based models across languages is crucial to ensure equitable access and accuracy of information in multilingual contexts. This study aimed to compare AI model efficiency in English and Arabic for infectious disease queries. Methods The study employed the METRICS checklist for the design and reporting of AI-based studies in healthcare. The AI models tested included ChatGPT-3.5, ChatGPT-4, Bing, and Bard. The queries comprised 15 questions on HIV/AIDS, tuberculosis, malaria, COVID-19, and influenza. The AI-generated content was assessed by two bilingual experts using the validated CLEAR tool. Results In comparing AI models' performance in English and Arabic for infectious disease queries, variability was noted. English queries showed consistently superior performance, with Bard leading, followed by Bing, ChatGPT-4, and ChatGPT-3.5 (P = .012). The same trend was observed in Arabic, albeit without statistical significance (P = .082). Stratified analysis revealed higher scores for English in most CLEAR components, notably in completeness, accuracy, appropriateness, and relevance, especially with ChatGPT-3.5 and Bard. Across the five infectious disease topics, English outperformed Arabic, except for flu queries in Bing and Bard. The four AI models' performance in English was rated as “excellent”, significantly outperforming their “above-average” Arabic counterparts (P = .002). Conclusions Disparity in AI model performance was noticed between English and Arabic in response to infectious disease queries. This language variation can negatively impact the quality of health content delivered by AI models among native speakers of Arabic. This issue is recommended to be addressed by AI developers, with the ultimate goal of enhancing health outcomes.

The Performance of OpenAI ChatGPT-4 and Google Gemini in Virology Multiple-Choice Questions: A Comparative Analysis of English and Arabic Responses

“…Similarly, Banimelhem and Amayreh reported suboptimal English to Arabic translation capabilities for ChatGPT [38]. Additionally, a recent study showed the superior performance of four generative AI models in English compared to Arabic in infectious disease queries [39], while an earlier study showed the inferior performance of ChatGPT in general health queries in Arabic dialects [35]. Additionally, the inferior performance of AI chatbots was reported in other non-English languages including Chinese [40], Polish [41], and Spanish [42].…”